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EXECUTIVE  SUMMARY 

The  need  for  effective  information  management  processes  has  become  more  urgent  as  the 
amount  of  information  available  has  increase  exponentially.  Information  that  formerly  had  to  be 
sent  through  the  mail  or  via  messengers  to  individual  recipients  can  now  be  disseminated  almost 
instantaneously  with  a  single  keystroke  to  huge  numbers  of  recipients  in  widespread  locations. 
Such  freedom  in  information  exchange  is  advantageous  in  that  more  people  are  more  able  to  find 
information  they  need  for  problem  solving  and  decision  making;  however,  one  disadvantage  of 
such  free  flowing  information  is  the  possibility  of  being  exposed  to  more  information  than  can  be 
comprehended,  a  state  known  as  information  overload. 

In  this  project  we  developed  a  model  of  information  management  that  was  derived  from 
sound  cognitive  theories  as  well  as  empirical  observations  of  individuals  attempting  to  manage 
information.  The  model  describes  the  process  by  which  incoming  information  is  filtered,  assessed, 
consumed  or  acted  upon,  and  disseminated,  and  information  gaps  are  filled  by  seeking  additional 
information.  In  this  model,  we  theorized  that  knowledge  of  the  organization  in  which  one  is 
operating  is  a  critical  aspect  of  efficient  information  generation;  it  is  only  with  such  organizational 
knowledge  that  information  disseminators  can  reduce  the  inadvertent  information  overload  caused 
,  by  thoughtless  information  propagation. 

Using  the  initial  model  and  observations  as  guides,  we  developed  an  information 
management  training  program  designed  to  help  information  managers  cope  effectively  and 
efficiently  with  large  volumes  of  incoming  information,  and,  at  the  same  time,  control  the  spread 
of  information  by  effective  and  efficient  information  generation.  Information  processing  activities 
generally  occur  in  the  context  of  some  planning,  problem  solving,  or  decision-making  task  in  an 
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organization;  for  this  project  we  oriented  the  training  to  information  management  in  military 
organizations. 

A  preliminary  version  of  the  information  management  training  program  focused  on  helping 
information  managers  to  define  their  information  needs  and  enhance  their  depth  of  knowledge. 

For  a  military  context,  we  focused  knowledge-based  training  on  increasing  their  understanding  of 
the  mission,  their  role  in  the  mission,  and  the  organizational  structure  in  which  the  mission  is 
occurring.  In  a  pilot  study  we  administered  the  training  program  to  a  group  of  cadets  at  the  United 
States  Military  Academy  at  West  Point.  Although  the  training  fostered  more  effective  high  level 
information  filtering,  it  did  not  help  participants  comprehend  and  integrate  the  incoming 
information.  We  concluded  that  the  program  needed  a  stronger  focus  on  behavioral  training  that 
emphasized  specific  information  management  techniques.  We  also  concluded  that  organizational 
knowledge  needed  to  be  incorporated  directly  into  the  information  management  training  program, 
rather  than  being  provided  as  a  separate  component,  as  was  done  in  the  pilot  study. 

Based  on  the  results  of  the  pilot  study,  we  developed  the  test  version  of  the  information 
management  training  program,  and  centered  it  around  a  five-step  behaviorally  oriented  approach  - 
Prepare,  filter,  scan,  read,  act  -  that  trainees  can  use  to  process  incoming  information.  We  set  the 
program  in  the  context  of  the  Military  Decision  Making  Process  (MDMP)  that  military  officers 
learn,  and  motivated  it  by  a  discussion  of  how  information  overload  occurs  and  the  cognitive  and 
psychological  results  that  it  imposes.  The  program  was  comprised  of  four  components  needed  for 
effective  training  -  lecture,  demonstration,  practice,  and  feedback. 

We  administered  the  training  program  to  mid-career  officers  at  the  Battle  Command  Battle 
Laboratory  at  Fort  Leavenworth,  and  assessed  its  effectiveness  by  comparing  the  information 
processing  performance  of  the  trained  subjects  to  that  of  a  control  group.  The  results  showed  the 
that  IM  training  program  was  successful  in  fostering  more  effective  information  management 
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processes,  and  in  mitigating  some  of  the  known  information  processing  errors  that  have  been 
observed.  Trained  subjects  were  more  effective  at  higher  level  filtering,  opening  more  critical 
and  fewer  nonrelevant  messages.  They  were  more  able  to  recognize  critical  messages  and  better 
able  to  integrate  the  information  in  the  messages  to  form  a  higher  quality  situation  assessment. 
Trained  subjects  were  also  more  effective  information  generators,  transmitting  fewer  but  more 
critical  messages,  and  sending  them  to  more  appropriate  and  more  specific  nodes. 

The  overall  evaluation  of  the  training  program  demonstrated  that  it  was  successful  in 
counteracting  many  of  the  typical  errors  that  occur  in  information  management.  The  individuals 
who  participated  in  the  training  evaluation  at  Fort  Leavenworth  are  mid-career  officers,  all  of 
whom  already  have  intensive  training  on  information  management,  and  many  of  whom  have  had 
extensive  experience  as  information  managers.  That  we  were  able  to  demonstrate  positive  effects 
of  a  two-hour  training  program  attests  to  its  effectiveness  and  promise. 

We  revised  the  information  management  model  that  had  served  as  a  basis  for  our  work  to 
to  incorporate  the  results  of  the  training  development  and  evaluation.  The  updated  model,  shown 
in  the  figure  below,  makes  explicit  the  importance  of  initial  preparation  in  information,  shows  how 
key  words  and  phrases  identified  in  preparation  can  be  used  in  scanning  messages  to  determine 
whether  they  contain  important  information,  better  depicts  the  dynamic  nature  of  information 
management,  and  emphasizes  the  importance  of  knowledge  of  the  organization  in  which  the 
information  processing  activities  are  occurring.  The  model  incorporates  a  feedback  loop 
emphasizing  that  information  management  is  a  dynamic  process  in  which  critical  information 
needs  and  sources  can  change  over  time.  It  also  shows  how  organizational  knowledge  impacts 
information  seeking  and  transmission. 
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CHAPTER  1 

DEVELOPMENT  AND  EMPIRICAL  INVESTIGATION  OF  AN  INFORMATION 

MANAGEMENT  MODEL 

In  this  report  we  describe  the  development,  implementation,  and  assessment  of  an 
information  management  (IM)  training  program.  The  focus  and  contents  of  the  training  program 
were  based  on  requirements  identified  in  an  IM  model  derived  from  empirical  observations  of 
decision  makers  in  various  domains  and  on  the  findings  from  an  empirical  investigation  of  IM 
under  varying  levels  of  information  load.  Detailed  reports  on  the  model  and  experiment  are  found 
in  Entin,  Kerrigan,  Serfaty,  Klein,  and  Wolf  (1996  and  1998,  respectively).  In  this  chapter  we 
summarize  the  IM  model  and  experiment,  with  a  focus  on  how  they  set  the  stage  for  the  IM 
training  program  detailed  in  the  following  chapters. 

Chapter  2  describes  the  initial  version  of  the  IM  training  program  we  developed  and 
evaluated.  Chapter  3  describes  the  design,  contents,  and  evaluation  of  the  second  version  of  the 
training  program.  Chapter  4  summarizes  our  findings,  reassesses  the  IM  model  in  light  of  the 
evaluations  we  conducted,  and  suggests  directions  for  future  work. 

1.1  MOTIVATION 

The  vision  of  the  Chairman,  Joint  Chiefs  of  Staff  for  warfare  in  the  information  age 
(Warfighting  Vision,  2010)  stresses  the  importance,  and  the  potential  problems,  of  the  information 
management  and  interconnectivity  implied  by  information  dominance  concepts  (Alberts,  1996), 
asserting  that  information  management  could  be  a  major  limitation  in  C4I  systems.  DiNardo  and 
Hughes  (1995)  recognized  data  overload  as  one  of  the  dangers  of  improved  communication 
technology,  because  commanders  can  become  so  overwhelmed  with  extraneous  information  that 
the  real  issues  can  be  obscured.  One  case  in  point  is  Desert  Storm,  where  the  Joint  Forces 
Commander  received  more  than  1.3  million  messages  during  the  first  24  hours.  To  ensure  that 
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staff  and  commanders  can  find  the  important  information,  it  is  essential  to  reduce  this  number  to  a 
more  manageable  level  in  future  conflicts. 

At  the  time  of  the  first  IM  experiment,  few  researchers  had  rigorously  investigated  how 
human  warfighters  could  best  take  advantage  of  the  potential  of  information-rich  networked 
command  and  control  organizations  without  becoming  overwhelmed.  We  know  that  a  critical 
component  of  military  command  expertise  is  the  ability  to  focus  on  important  information,  identify 
missing  information,  and  “ask  the  right  questions”  when  assessing  a  situation  (Serfaty,  MacMillan, 
Entin,  and  Entin,  1996),  but  it  is  not  clear  how  someone  lacking  this  expertise  can  best  learn  these 
behaviors.  A  goal  of  the  current  research  program  was  to  develop  a  training  program  to  facilitate 
IM  for  the  non-expert. 

1.2  DEFINITION  AND  MODEL  OF  INFORMATION  MANAGEMENT 

For  the  purposes  of  this  study,  IM  was  defined  as  a  set  of  cognitive  processes  and 

behaviors  that  include  the 
receiving,  filtering, 
integrating,  processing, 
seeking,  and  exchanging  of 
information  by  an  individual 
decision  maker  in  an 
organization.  Figure  1-1 
shows  the  problem  of  IM 
from  the  perspective  of  the 

information  receiver.  The  information  manager  has  a  finite  amount  of  information  needs  -  that 
information  required  for  effective  performance  -  but  receives  much  more  information  from 


Figure  1-1.  Information  Management  for  One  DM 
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external  sources  such  as  the  environment  or  the  structure  of  the  organization  than  is  required. 
Despite  this  massive  information  push,  there  will  still  be  information  that  the  information  manager 
must  seek.  The  ability  to  filter  information  and  recognize  information  gaps  is  becoming 
increasingly  important  as  the  information  dissemination  technology  continues  to  develop  faster 
than  automatic  information  filters.  The  ensuing  free-flow  of  information  results  in  a  flatter 
command  and  control  (C2)  organization  in  which  everyone  has  access  to  more  data  than  ever 
before.  Those  who  cannot  manage  information  succumb  to  information  overload  (See  Figure  1-2). 
The  goal  of  this  program  was  to  develop  theory-driven  training  methods  that  will  enhance 
effective  IM  skills  and  practices,  allowing  individuals  to  effectively  filter  incoming  information 
and  focus  on  needed  rather  than  surplus  information,  and,  at  the  same  time,  to  recognize  their 
critical  information  needs,  and  seek  that  information  from  the  appropriate  sources. 


Increase  Access  to  Information 
Internet,  Intranets,  Digitization  of  the  Battlefield 


Figure  1-2.  Information  Overload  in  C2  Organizations 


Given  the  definition  that  IM  is  a  set  of  cognitive  processes,  a  cognition-based  model  of  IM  was 
developed  (see  Figure  1-3)  and  used  to  guide  the  experimental  design  as  well  as  the  development 
and  use  of  innovative  information  flow  measures  to  capture  the  key  processes  underlying  IM  in  a 
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C2  organization.  This  model  of  IM  is  based  on  observations  of  decision  makers  in  more  than  20 
different  domains  including  urban  and  wildland  firefighters,  Army  and  Marine  tactical  operation 
center  personnel,  JFACC  planners,  Army  air  defense  planners,  Navy  commanders,  Navy  combat 
information  center  staff,  and  many  others.  Each  of  these  domains  is  characterized  by  decision 
making  under  high  stakes,  high  stress,  and  high  time  pressure,  using  limited  information  from 
multiple  sources.  The  resultant  model  is  not  complicated,  it  can  be  viewed  in  simplest  terms  as: 
receive  some  set  of  information,  determine  what  that  information  means,  and  act  on  it.  Other  key 
aspects  of  this  IM  model  are  the  two  levels  of  filtering  necessary  for  good  IM  skills.  The  first,  or 
“higher  level,”  filter  can  be  viewed  as  a  simple  gate  that  is  set  to  reject  or  accept  the  information 
for  further  processing  based  on  immediately  available  characteristics  (e.g.,  source  and  subject), 
whereas  the  second,  or  “low  level,”  filter  is  based  on  a  more  complete  processing  of  the 
information  (e.g.,  reading  the  entire  message). 


Figure  1-3.  Conceptual  Model  of  Information  Management 
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As  with  any  cognitive  process,  errors  are  typically  an  integral  part  of  managing  information  and 
can  occur  at  any  step  in  the  IM  process;  Figure  1-4  illustrates  the  most  common  errors  seen  in  IM. 
For  example,  errors  can  occur  at  the  high  level  filter  when  the  information  manager  decides  to 
pursue  irrelevant  information  or  ignores  important  information,  or  at  the  low  level  filter  if  the 
information  manager  fails  to  disregard  obsolete  information,  or  becomes  distracted  by  interesting 
information  that  is  not  relevant  for  the  mission  at  hand.  There  can  also  be  errors  in  information 
transmission  and  information  seeking  behaviors. 


>|  Filter 


Pursue  wrong  information 
Disregard  infonnation 


passinc  data  or  information 

T 

Transmit  data,  information 
or  understanding  to  other 
components  of  the  system 


Figure  1-4.  Potential  Errors  in  the  Information  Management  Process 
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1 3  EMPIRICAL  ASSESSMENT  OF  INFORMATION  MANAGEMENT 

Based  in  the  IM  model  and  the  associated  IM  processing  errors  that  have  been  observed, 
we  developed  and  conducted  an  experiment  to  validate  the  cognitive  model  of  information 
management  including  the  possible  errors,  and  to  empirically  identify  IM  strategies,  techniques, 
and  skills  amenable  to  enhancement  through  training  that  can  prevent  these  errors  from  happening. 
In  particular,  we  sought  to  identify  skills  that  were  especially  vulnerable  under  conditions  of  high 
information  load,  and  to  develop  strategies  to  mitigate  the  deleterious  effects 
of  information  overload. 

The  two  major  independent  variables  implemented  in  this  experiment,  information  load  and 
knowledge  of  organization  structure.  Information  load  was  implemented  as  a  within-subjects 
variable  with  two  levels,  moderate  and  high.  In  the  moderate  information  load  condition  there  was  a 
reasonable  rate  of  information  flow,  whereas  in  the  high  information  load  condition  the  individual 
was  deluged  with  information.  The  information  necessary  for  understanding  and  assessment  of  the 
evolving  situation  was  provided  in  both  the  high  and  moderate  load  conditions.  Knowledge  of 
organizational  structure  was  implemented  as  a  between-subjects  variable  with  two  levels,  shallow 
and  deep.  For  shallow  knowledge,  we  provided  a  diagram  of  the  organizational  structure  and  a  one- 
line  description  of  each  available  node.  For  deep  knowledge,  we  supplemented  the  diagram  with  a 
detailed  written  description  of  the  roles  and  functions  of  each  node,  and  the  classes  of  information 
possessed  and  required  by  each  node.  We  used  an  unconventional,  futuristic  organization  so 
participants  could  not  easily  tap  into  existing  knowledge. 

The  major  performance  tasks  were  periodic  situation  assessment  briefings,  both  oral  and 
written.  Workload  was  assessed  using  the  The  Task  Load  Index  (TLX;  Hart  and  Staveland,  1988). 
Several  self-report  measures  were  developed  based  on  participants’  responses  to  items  in  Post-Trial 
and  Post-Experiment  Questionnaires. 
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Participants  were  required  to  rate  the  importance  of  all  incoming  messages  that  they 
opened  on  a  4-point  scale  ranging  from  irrelevant  to  highly  critical.  Participants  were  also 
permitted  to  send  messages  to  different  nodes  in  the  organizational  structure  to  ask  for  more 
information  about  the  situation,  or  to  forward  messages  to  another  node.  Two  dependent  measures 
were  developed  for  messages  sent  out  by  participants:  node  level  (subordinate,  lateral  and 
superior)  and  node  appropriateness  (was  the  message  sent  to  an  appropriate  or  inappropriate  node). 

The  basic  scenario  is  set  in  the  year  2008.  The  United  States  is  leading  a  multi-national, 
combined  task  force  (CTF)  to  implement  a  United  Nations  resolution  condemning  a  Country  A’s 
invasion  of  its  neighbor,  Country  B.  The  scenario  materials  included:  a  basic  introduction  to  the 
scenario  describing  the  geopolitical  situation  and  the  events  leading  up  to  the  current  situation; 
a  wall  map  showing  a  detailed  view  of  the  general  area  in  which  the  scenario  is  occurring;  two 
vignettes  based  on  the  general  scenario;  a  smaller  desk  map  showing  a  narrower  view  of  the  tactical 
situation  for  each  vignette,  plus  two  overlays  for  the  desk  map  showing  changes  in  the  tactical 
situation,  and;  a  stream  of  electronic,  telephone,  and  handwritten  messages  associated  with  each 
vignette,  some  of  which  are  viewed  as  critical  to  a  situation  assessment  and  others  of  which  do  not 
contain  essential  information. 

The  Distributed  Dynamic  Decision-making  (DDD)  simulator  (Serfaty  and  Kleinman,  1985; 
Kleinman  and  Serfaty,  1989)  was  used  as  the  experiment  testbed.  It  was  modified  to  include  a  message¬ 
sending  capability  whereby  a  set  of  preplanned  messages  could  be  incorporated  into  the  scenario  and 
sent  out  as  electronic  messages  at  specified  times  in  the  execution  of  a  scenario.  Other  new  features 
incorporated  into  the  DDD  simulator  include  a  message  listing  panel,  which  displays  external  message 
markers  such  as  the  time  and  subject  of  a  message,  and  a  mechanism  for  rating  both  incoming  and 
outgoing  messages. 
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Sixteen  military  offers  participated  in  the  experiment.  Because  the  amount  of  time 
available  for  experiment  sessions  was  limited,  participants  were  given  the  background  scenario 
materials  two  weeks  prior  to  the  experiment  and  asked  to  read  and  review  all  materials  they 
received  before  participating  in  the  first  experiment  session.  Figure  1-5  provides  a  high-level 
overview  of  the  experiment  design  and  procedure.  Each  participant  participated  individually  in 
two  experiment  sessions. 


Brief  Brief  Brief  Brief  Brief  Brief 

Figure  1-5  Overview  of  Experiment  Design  and  Procedure 


Analysis  of  the  workload  ratings  confirmed  that  subjects  experienced  a  significantly  higher 


workload  in  the  high  information  load  condition.  The  results  revealed  that  participants  could  more 
accurately  assess  the  situation  when  exposed  to  a  moderate  level  of  information  load  than  when 
exposed  to  a  high  information  load.  The  problems  exacerbated  by  high  level  of  information  load 
were:  1)  difficulty  recognizing  critical  information  and  2)  difficulty  in  filtering  out  noncritical 
information.  Figure  1-6  provides  a  summary  of  the  errors  exhibited  by  the  participants  in  this  first 
experiment.  Specifically,  it  was  found  that  participants  had  a  difficult  time  formulating  and 
directing  messages  -  too  many  messages  (many  inappropriate)  were  sent  to  subordinates  and  the 


messages,  if  forwarded  without  change,  were  more  likely  to  be  misdirected. 
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Figure  1-6.  Observed  Errors  and  Difficulties  in  Information  Management 
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Evidence  also  suggested  that  a  deeper  level  of  organizational  knowledge  facilitated  the 
transfer  of  information  to  the  appropriate  nodes  in  the  organization  when  information  load  was 
high,  although  the  exact  effect  of  deep  organizational  knowledge  was  attenuated  because  most 
participants  indicated  on  the  post  experiment  questionnaire  that  they  did  not  have  sufficient  time 
prior  to  the  experiment  to  read  through  all  of  the  pre-experiment  packet. 

1.4  TRAINING  IMPLICATIONS 

Based  on  the  experiment  and  other  empirical  findings,  we  identified  two  high  level 
objectives  for  an  IM  training  program  designed  to  enhance  effective  IM  skills  and  practices: 

1)  Training  associated  with  the  identification  of  one’s  own  information  requirements  to  help 
decision-makers  recognize  their  critical  information  needs  and  focus  on  needed  rather  than 
surplus  information,  and;  2)  Enhancement  of  organizational  knowledge  to  provide  decision 
makers  with  a  clearer  understanding  of  the  needs  and  capabilities  of  other  nodes  in  the 
organization  and  the  lines  of  communication  among  the  nodes,  and  thereby  facilitate  effective  and 
efficient  information  interchange  among  the  nodes. 

The  proposed  approach  for  implementing  the  training  program  suggested  that  some  of  the 
training  should  focus  on  the  knowledge  of  the  individuals,  specifically  defining  information  needs, 
and  enhancing  depth  or  level  of  knowledge.  The  most  important  skill  the  subjects  in  the 
experiment  seemed  to  be  missing  was  the  ability  to  recognize  critical  information  (i.e.,  define 
critical  needs).  It  was  concluded  that  participants  need  focused  understanding  of  the  mission,  of 
their  role  in  the  mission,  and  of  the  tasks  they  are  called  upon  to  perform  to  fulfill  their  role  in  the 
mission.  The  approach  also  suggested  that  enhancing  depth  or  level  of  organizational  knowledge 
is  another  important  area  for  increased  knowledge  since  the  original  study  found  some  support  that 
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an  increase  in  organizational  knowledge  helped  counteract  the  effects  of  massive  amounts  of 
information. 

The  empirical  results  suggested  that  participants  need  to  have  some  procedural  training  in 
addition  to  the  knowledge  training.  It  was  apparent  in  the  experiment  that  not  only  did  the  subjects 
not  know  where  to  send  or  get  messages,  but  they  did  not  know  how  to  send  or  get  messages.  It 
was  proposed  that  participants  needed  training  in  how  and  where  to  send  information  (information 
push)  and  how  and  where  to  get  information  (information  pull). 
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CHAPTER  2 

INFORMATION  MANAGEMENT  TRAINING  PROGRAM:  DEVELOPMENT  AND 
ASSESSMENT  OF  A  PRELIMINARY  VERSION 

Based  on  the  information  management  (IM)  model  and  experiment  findings  described  in 
the  previous  chapter,  the  ALPHATECH/ Aptima  team  developed  a  preliminary  version  of  an  IM 
training  program.  In  this  chapter  we  describe  this  initial  version  and  the  conduct  of  a  pilot  study  to 
explore  the  appropriateness  of  that  program.  The  results  from  the  pilot  study  were  intended  to  be 
used  in  the  further  development  of  the  training  program.  Section  2.1  contains  the  description  of 
the  content  and  format  of  the  training  program;  Section  2.2  explains  the  materials  and  method  we 
used  to  assess  the  appropriateness  and  effectiveness  of  the  program;  Section  2.3  describes  the  pilot 
evaluation  that  we  conducted;  and  Section  2.4  discusses  the  conclusions  and  implications  we  drew 
from  the  pilot  investigation. 

2.1  DEVELOPMENT  OF  TRAINING  PROGRAM 

2.1.1  Description  of  the  Training  Program 

Based  on  the  findings  of  the  Experiment  1  (described  in  Chapter  1),  a  preliminary  IM 
Training  Program  was  developed  to  enhance  two  knowledge  bases:  defining  information  needs 
and  enhancing  depth  or  level  of  knowledge.  For  information  managers  working  in  a  military 
context,  we  hypothesized  that  knowledge-based  training  focused  on  increasing  their  understanding 
of  the  mission,  their  role  in  the  mission,  the  tasks  they  are  called  upon  to  perform  to  fulfill  their 
role  in  the  mission,  as  well  as  the  organizational  structure  in  which  their  mission  is  occurring  will 
help  them  deal  with  massive  amounts  of  information.  In  effect,  the  training  program  attempted  to 
help  information  processors  wade  through  massive  amounts  of  information,  focusing  on  the 
relevant  information  while  disregarding  irrelevant  information.  The  actual  training  program  is 
described  below.  The  training  was  conducted  and  evaluated  at  the  United  States  Military 
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Academy  at  West  Point  with  upper-class  cadets  enrolled  in  a  systems  engineering  class.  It  is 
important  to  note  that  the  style  and  content  of  the  training  materials  was  developed  with  this 
audience  in  mind. 

Participants  in  the  IM  Training  Program  received  extensive  training  (lecture-style  instruction 
and  a  written  demonstration)  aimed  at  increasing  their  ability  to  assess  the  situation,  be  more  aware  of 
time  horizons,  and  help  them  handle  uncertainty  and  time  constraints  using  the  mnemonic  MISSION. 
The  M  in  Mission  corresponds  with  “delineate  Mission;”  the  I  corresponds  with  “Identify  critical 
elements  of  information;”  the  first  S  corresponds  with  “Specify  messages  that  could  help  you;”  the 
second  S  corresponds  with  “Speculate  how  you  might  help  others;”  the  second  I  corresponds  with 
“Ignore  irrelevant  information;”  the  O  corresponds  with  “Observe  time  constraints;”  and  the  N 
corresponds  with  “keep  a  Narrow  focus.”  We  believed  that  learning  this  mnemonic  would  help 
inexperienced  information  processors  and  decision-makers,  such  as  the  West  Point  cadets,  focus  on 
the  relevant  information  while  disregarding  the  irrelevant  information.  In  other  words,  it  would  help 
the  novice  become  more  expert  in  identifying  and  processing  critical  information  and  subsequently  in 
making  effective  decisions. 

Information  management  training  was  implemented  in  a  lecture/demonstration  format.  For 
the  lecture  portion  subjects  received  a  booklet  and  followed  along  with  the  trainer  (Figure  2-1  is 
the  cover  sheet  for  the  Information  Management  manual).  The  subjects  were  told  that  the  purpose 
of  the  training  was  to  help  them  sift  through  large  amounts  of  information,  determining  what  is 
important  and  what  is  not.  It  was  explained  to  them  that  there  are  a  number  of  things  that  must  be 
done  to  effectively  deal  with  large  amounts  of  information  and  that  the  manual  being  used  for  the 
training  breaks  them  down  into  the  following  seven  parts: 
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Figure  2-1.  Cover  for  Training  Manual 
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For  each  of  these  seven  parts  the  manual  contained  a  thorough  explanation  about  what  is 
meant  (labeled  “WHAT  IS  THIS?”  in  the  manual),  an  example  scenario  to  show  how  it  is 
(“HOW  DO  I  DO  THIS?”),  and  an  opportunity  for  the  subjects  to  practice  with  another  scenario 
(“APPLICATION.”)-  It  was  further  explained  that  although  the  parts  were  explained  sequentially, 
they  are  not  steps  to  be  performed  in  strict  order;  they  should  all  be  thought  about 
at  the  start  of,  and  throughout,  every  mission. 

After  receiving  the  initial  IM  training,  all  subjects  were  exposed  to  a  written 
“demonstration”  designed  to  further  strengthen  the  mnemonic  MISSION.  In  this  demonstration, 
subjects  read  a  description  of  a  fictitious  incident  broken  up  into  seven  parts  (corresponding  to  the 
seven  parts  described  in  the  training  lecture  portion  described  above).  Each  part  was  described 
twice  with  the  first  depiction  illustrating  an  incorrect  way  to  handle  the  situation,  and  the  second 
depiction  illustrating  a  better  way  to  handle  things.  As  a  group  the  trainer  and  the  participants 
read  the  scenes  as  presented  and  answered  questions  designed  to  strengthen  the  participants’ 
ability  to  focus  on  their  MISSION. 

2.1.2  Description  of  Control  Training 

To  evaluate  the  effectiveness  of  any  training  program  it  is  necessary  to  have  a  control 
group  of  subjects,  who  do  not  receive  the  training  but  receive  some  comparable  instruction-type 
interaction,  against  which  to  compare  the  trained  subjects.  To  provide  a  counterpart  to  the  IM 
training  program  we  developed  another  set  of  materials,  which  we  refer  to  as  the  control  training, 
that  we  used  with  the  control  group.  This  control  training  was  implemented  in  a  similar  lecture 
format  as  that  used  for  the  IM  training. 

Because  the  scenario  they  would  be  participating  in  was  set  in  the  future  (2008),  the 

control  training  focused  on  the  Army  of  the  future  since  such  training  could  be  seen  as  “setting  the 

stage”  for  the  experimental  scenario.  Similar  to  the  IM  training,  the  control  training  was  divided 
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into  two  parts,  the  first  of  which  focused  on  how  the  changes  in  the  Army  might  affect  the  subjects 
personally  (Figure  2-2  is  the  cover  for  the  control  training  manual),  then  the  second  part  was  a 
more  general  overview  of  the  Army  After  Next.  It  was  explained  that,  since  allied  forces  were 
joint  in  the  experimental  scenario,  some  background  on  the  vision  for  the  Army  After  Next  would 
set  the  stage  for  the  exercise. 


Figure  2-2.  Cover  for  Control  Manual 
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2.2  TRAINING  EVALUATION:  METHOD 

In  this  section  we  describe  the  scenario  materials  used  in  the  evaluation  of  the  training  and 

the  experiment  design  and  procedure  that  was  employed. 

2.2.1  Scenario  Materials 

We  used  the  same  basic  scenario  as  was  used  in  the  information  load  experiment  described 
in  Chapter  1 .  The  scenario  is  set  in  the  year  2008.  The  United  States  is  leading  a  multi-national, 
combined  task  force  (CTF)  to  implement  a  United  Nations  resolution  condemning  an  Country  A’s 
invasion  of  its  neighbor,  Country  B.  The  CTF  Commander  (CTFC)  has  established  an 
organizational  structure  that  uses  four  functional  component  commanders  for  air,  land,  maritime, 
and  special  operations  joint  task  forces.  The  scenario  materials  used  for  the  evaluation  of  the  IM 
training  included  a  basic  introduction  to  the  scenario  describing  the  geopolitical  situation  and  the 
events  leading  up  to  the  current  situation,  and  a  large  annotated  map  showing  a  detailed  view  of 
the  geographic  area  in  which  the  scenario  was  set.  A  Situation  Update  bridged  from  the  time 
described  in  the  basic  scenario  to  the  current  situation. 

The  basic  scenario  sets  the  stage  for  the  vignette  that  was  used  in  the  experiment.  In  the 
vignette,  the  participant  is  part  of  the  maritime  component,  and  plays  the  role  of  a  member  of  the 
Force  I  commander’s  staff,  located  on  an  amphibious  command  and  control  ship  in  the  region. 

The  mission  is  to  conduct  a  situation  assessment  and  determine  why  enemy  armor  units  have 
halted  in  the  vicinity  of  Objective  A  (labeled  on  the  annotated  map).  The  map  was  used  in  the 
experiment  as  a  means  of  conveying  graphical  information  was  a  detailed  road  map  of  the  two 
countries  involved.  The  annotations  on  the  map  showed  the  locations  of  enemy  and  friendly  units 
as  well  as  named  areas  of  interest. 


Training  Information  Management 
In  Distributed  Organizations 


17 


ALPHATECH,  INC. 


2.2.2  Organizational  Knowledge 

Declarative  knowledge  about  the  structure  and  functions  of  an  organization  and  dynamic 
knowledge  about  how  the  structure  is  used  in  a  particular  situation  are  two  aspects  of 
organizational  knowledge.  Declarative  knowledge  includes  information  about  the  structure  of  the 
organization,  communication  channels,  role  and  functions  of  individuals  within  the  organization, 
and  information  sources.  We  hypothesized  that  participants  with  deep  declarative  knowledge  of 
organizational  structure  will  have  a  better  framework  for  applying  aspects  of  the  IM  training 
concerned  with  the  sources  of  information,  the  need  to  forward  information,  and  the  subjects  of 
that  information. 

In  the  evaluation  experiment,  there  were  two  levels  of  organizational  knowledge:  shallow 
and  deep.  To  implement  shallow  knowledge,  we  provided  a  one-page  summary  showing  a  diagram 
of  the  organizational  structure  and  a  one-line  description  of  each  available  node.  For  deep 
knowledge,  we  supplemented  the  diagram  with  a  detailed  written  description  of  the  roles  and 
functions  of  each  node,  the  classes  of  information  possessed  and  required  by  each  node,  and  a 
description  of  the  information  flow  between  the  nodes.  We  used  an  unconventional,  futuristic 
organization  so  participants  could  not  easily  tap  into  existing  knowledge  of  a  traditional  military 
organizational  structure.  The  Organizational  Chart  given  to  the  subjects  is  included  in  Appendix  A. 
The  materials  distributed  in  the  shallow  organizational  knowledge  condition  (the  organizational 
chart  and  a  one-line  functional  description  for  each  position)  are  also  included  in  Appendix  A. 

We  sent  a  reading  packet  to  the  participants  in  the  study  ahead  of  time  so  they  could  leam 
the  structure  of  the  fictional  organizational  before  going  to  the  study.  The  material  was  divided 
into  small,  manageable,  segments  after  each  of  which  there  was  a  self-test  to  help  the  participant 
leam  the  material.  The  answers  for  the  self-tests  were  at  the  back  of  the  booklet. 
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2.2.3  Scenario  Generator 

The  scenario  generator  used  for  the  training  evaluation  was  a  tailored  version  of  a  JAVA- 
based  mail  message  software  package  implemented  for  a  PC  running  Windows  NT.  The  program 
was  capable  of  sending  messages  out  at  pre-specified  times.  It  could  also  transmit  messages  sent  by 
the  subjects  to  other  nodes  in  the  organization  and  transmit  back  to  the  subject  a  response 
formulated  by  the  controller,  who  simulated  all  other  nodes. 

An  example  of  the  main  display  is  shown  in  Figure  2-3.  The  message  queue  lists  the 
messages  that  have  been  received.  Unopened  messages  are  listed  in  black  and  messages  that  have 
been  opened  are  listed  in  grey.  To  read  a  message  the  participant  clicks  on  the  message,  and  it 
appears  in  the  message  listing  window.  To  send  a  message,  the  participant  clicks  on  the  New 
Message,  Forward,  or  Reply  button  located  above  the  message  listings. 

Figure  2-4  shows  an  example  of  the  message  window,  which  includes  the  message  header, 
the  source,  and  the  time  of  the  message,  the  message  body.  The  bottom  of  the  window  shows  the 
rating  scale  used  by  the  participant  to  evaluate  the  criticality  of  the  message.  Figure  2-5  shows  an 
example  of  the  message  generation  window,  which  includes  place  holders  for  the  intended 
recipient,  the  subject,  and  the  body  of  the  message. 

2.2.4  Instruments 

Five  instruments  were  used  during  the  evaluation  experiment,  the  first  four  of  which  were 
completed  by  the  subjects,  and  the  fifth  by  an  SME  evaluator.  (See  Appendix  A  for  a  copy  of  each 
of  the  instruments.) 

Organizational  Knowledge  Questionnaire 

This  questionnaire  was  used  to  assess  subjects’  knowledge  of  the  organizational  structure 
and  the  roles  and  information  requirements  of  the  nodes  within  the  organization. 
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Figure  2-5.  Sample  of  Editor  Window  Showing  Scrollable  List  of  Recipients 


Briefing  Preparation  Form 

The  Briefing  Preparation  Form  was  designed  to  help  subjects  prepare  the  interim  and  final 
oral  briefings.  It  was  structured  around  four  questions  pertaining  to  uncertainty,  potential  sources 
of  information,  and  current  situation  in  the  vignette. 

Workload  Questionnaire 

The  Task  Load  Index  (TLX;  Hart  and  Staveland,  1988)  is  a  self-report  measure  that  elicits 

a  participant’s  ratings  on  six  dimensions  of  workload  (mental  demand,  physical  demand,  temporal 

demand,  performance,  effort,  and  frustration).  The  TLX  appears  to  enjoy  high  validity  and 

reliability  (Lysaght  et  al.,  1989).  Participants  respond  using  a  20-point  graphical  scale  anchored  at 

one  end  by  the  words  “very  low”  and  at  the  other  end  by  the  words  “very  high.”  For  this  study,  the 
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mean  of  each  participant’s  responses  to  five  of  the  six  dimensions  (physical  demand  was  excluded 
because  it  was  not  germane  to  the  task  being  performed)  was  used  as  the  primary  measure  of 
workload. 

Post-Experiment  Questionnaire 

The  post-experiment  questionnaire  was  a  self-report  measure  that  probed  participants’ 
views  about  the  timing  of  the  experiment  and  asked  them  to  assess  the  value  of  the  training  they 
received.  (See  Appendix  A  for  the  post-experiment  questionnaire.) 

Situation  Assessment  Evaluation  Form 

The  SME  who  rated  the  subjects’  written  and  oral  briefings  used  this  form  to  assess  aspects 
of  the  subjects’  briefings  and  provide  an  overall  rating  of  the  briefing. 

2.2.5  Experiment  Design 

The  primary  purpose  of  the  experiment  was  to  test  the  effectiveness  of  the  IM  training 
program  we  developed.  It  also  assessed  the  impact  of  depth  of  knowledge  of  organization 
structure.  Three  systems  engineering  classes  at  the  Military  Academy  participated  in  the  training 
program.  The  number  of  subjects  available  precluded  a  fully  crossed  evaluation  design.  Subjects 
were  assigned  to  one  of  three  experiment  conditions:  1)  IM  training  with  deep  organizational 
knowledge;  2)  IM  training  with  shallow  organizational  knowledge;  and  3)  control  training  with 
shallow  organizational  knowledge. 

Because  of  an  interest  expressed  by  some  members  of  the  Systems  Engineering  faculty  at 
the  Academy  in  the  impact  of  spoken  versus  written  messages,  we  also  varied  the  type  of  messages 
that  were  sent  out.  In  each  of  the  three  experiment  conditions,  half  the  subjects  received  all 
messages  in  written  format.  The  other  half  received  about  60  percent  of  the  messages  in  the  written 
format  and  about  40  percent  as  oral  messages. 
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2.2.6.  Participants 

Thirty-six  upper  class  cadets  at  the  Military  Academy  at  West  Point  participated  in  the 
experiment.  The  cadets  participated  in  conjunction  with  a  systems  engineering  class  in  which  they 
were  enrolled.  Three  classes  of  students  were  involved  and  the  students  were  assigned  to 
treatment  condition  by  class.  One  class  of  16  students  received  the  IM  training  program  and  the 
deep  organizational  knowledge  materials.  A  second  class  of  1 1  students  received  the  IM  training 
program  and  the  shallow  organizational  knowledge  materials.  A  third  class  of  9  students  received 
the  control  training  and  the  shallow  organizational  knowledge  materials. 

2.2.7  Procedure 

Three  experimenters  were  involved  in  the  training  and  evaluation.  They  conducted  all 
aspects  of  the  training  program  and  administered  all  written  and  oral  questionnaires  during  the 
evaluation.  One  of  the  three  served  as  the  primary  trainer  who  conducted  the  lecture  portion  of 
the  training  program.  The  three  experimenters  each  conducted  one  of  the  three  demonstration 
segments.  Three  instructors  in  the  systems  engineering  department  supported  the  conduct  of 
the  experiment  by  serving  as  controllers.  Their  duties  involved  role  playing  other  nodes  in  the 
organization  by  responding  to  questions  that  participants  posed  in  their  electronic  messages, 
listening  to  the  subjects’  oral  situation  assessment  briefings,  and  assisting  the  primary 
experimenter  in  handing  out  and  collecting  forms  and  questionnaires  to  the  subject. 

Because  the  amount  of  time  available  for  experiment  sessions  was  limited,  some  materials 
needed  for  the  training  evaluation  were  distributed  ahead  of  time.  Participants  were  given  the 
background  scenario  materials  and  the  materials  describing  the  organizational  structure  one  week 
prior  to  the  training  evaluation.  These  materials  included  background  information  about  the 
situation  that  set  the  stage  for  the  particular  situation  used  in  the  experiment,  and  information 
about  the  organizational  structure  in  which  they  would  be  working.  For  the  materials  associated 
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with  organizational  structure,  two  classes  of  students  received  the  materials  associated  with 
shallow  knowledge  of  organizational  structure  and  one  class  received  the  materials  associated  with 
deep  knowledge  of  organizational  structure.  All  participants  were  asked  to  carefully  read  and 
review  all  materials  they  received  before  participating  in  the  experiment  session.  They  were  also 
asked  not  to  discuss  the  materials  with  anyone  else  until  the  entire  experiment  was  completed. 

The  conduct  of  the  training  and  evaluation  experiment  was  carried  out  over  three  class 
periods.  In  the  first  class  period,  after  the  trainer  had  introduced  herself,  all  students  completed 
the  Organizational  Knowledge  Questionnaire  designed  to  assess  their  level  of  knowledge  about  the 
organization.  They  also  signed  a  consent  form  indicating  their  willingness  to  participate  in 
the  experiment.  The  remainder  of  the  class  period  was  devoted  to  the  lecture  portion  of  the 
training.  The  trainer  presented  the  training  concepts,  and  the  students  followed  along  in  a 
specially  prepared  booklet.  Two  of  the  three  participating  classes  received  the  IM  training 
program.  The  third  class  received  the  control  training.  At  the  end  of  the  class  hour,  the  trainer 
provided  a  preview  of  the  activities  that  would  occur  in  the  second  class  period. 

The  second  class  period  was  used  for  the  demonstration  portion  of  training  and  for  training 
in  the  use  of  the  scenario  generator  to  be  used  in  the  experiment.  During  the  demonstration 
segment  of  the  training,  which  took  approximately  half  of  the  50-minute  period,  a  trainer  discussed 
with  the  students  example  situations  that  illustrated  the  concepts  covered  in  the  lecture  portion  of 
the  training.  For  example,  the  demonstration  materials  showed  examples  of  appropriate  ways  to 
handle  messages  that,  while  important  in  a  real-world  context,  did  not  bear  upon  the  subject’s 
mission. 

Following  the  demonstration,  which  occurred  in  the  regular  classroom,  the  trainer  then 
handed  out  the  “op  order”  which  explained  the  current  situation  for  the  scenario  and  gave  tasking 
to  the  Force  I  commander.  The  students  were  urged  to  read  it  over  carefully  before  the  third  class 
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period.  The  students  were  then  directed  to  the  systems  engineering  computer  laboratory,  where 
they  received  training  on  how  to  use  the  messaging  software  that  would  be  used  in  the  experiment 
for  receiving,  reading,  rating,  and  sending  messages.  One  of  the  experimenters  explained  the 
processes  that  were  involved  to  the  students,  who  were  then  given  an  opportunity  to  try  out  the 
software  and  to  ask  any  questions  they  had  about  how  to  use  it.  Before  departing  students  were 
given  instructions  on  how  to  fill  out  the  Task  Load  Index  (TLX)  questionnaire  that  would  be 
administered  at  the  end  of  the  experiment  trial.  They  were  also  given  a  brief  overview  of  the  plan 
for  the  third  class  period  during  which  the  evaluation  of  the  training  was  to  be  conducted. 

On  entering  the  laboratory  for  the  trial  assessing  the  effectiveness  of  the  training  program, 
participants  sat  down  at  one  of  the  terminals  and  were  introduced  to  the  member  of  the  experiment 
team  who  would  serve  as  the  controller  for  them.  Their  attention  was  also  directed  to  a  large  map 
of  the  geographic  region  in  the  scenario.  There  were  three  maps  mounted  on  easels  in  the  front, 
middle,  and  back  of  the  room  to  make  them  easily  accessible  to  the  subjects  during  the  experiment 
trial. 

One  of  the  experimenters  then  gave  a  brief  capsule  review  of  the  scenario,  and  gave  the 
subjects  instructions  on  what  they  needed  to  do  during  the  experiment  trial.  Subjects  then  received 
written  information  about  the  vignette  associated  with  the  trial.  The  vignette  introduced  the 
specific  mission  in  which  the  subject  was  playing  and  the  subject’s  information  processing  task. 
Once  the  background  information  for  the  vignette  was  given,  the  message  processing  system  was 
started  and  incoming  electronic  messages  started  to  arrive. 

Figure  2-6  provides  a  high-level  overview  of  the  experiment  design  and  procedure.  The 
experiment  trial,  itself,  lasted  30  minutes.  During  the  experiment  trial,  subjects  had  the  capability 
to  read,  save,  delete,  list,  and  send  electronic  messages,  either  to  transfer  information  to  nodes 
more  equipped  to  deal  with  a  received  message  or  to  request  information  from  another  node  in  the 
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organization.  Subjects  in  the  experiment  played  the  role  of  a  member  of  the  Force  I  commander’s 
staff.  All  other  nodes  in  the  organization  were  role  played  by  a  controller,  who  responded  to 
participants’  mail  messages  when  appropriate. 
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Figure  2-6.  Overview  of  Experiment  Design  and  Procedure 
The  experiment  trial  was  divided  into  two  15-minute  intervals.  At  the  end  of  the  first  15- 
minute  period,  the  simulator  was  paused.  At  this  time,  participants  responded  to  the  four  questions 
on  the  written  briefing  preparation  form  and  then  gave  an  oral  briefing  based  on  this  written 
material. 

After  the  interim  data  collection  period,  the  simulation  was  resumed,  with  participants 
continuing  to  process  messages  and  assess  the  situation.  At  the  end  of  the  second  15-minute  period, 
participants  completed  a  written  situation  assessment  and  briefed  their  assessment  the  situation  to 
their  superior,  played  by  the  controller.  Participants  were  afforded  more  time  to  compose  the  final 
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briefing  than  the  interim  briefing.  The  two  oral  situation  assessment  briefings  given  by  the 
participants  were  tape  recorded.  After  the  second  time  period,  in  addition  to  preparing  and  delivering 
a  briefing,  subjects  completed  the  TLX  workload  questionnaire.  They  were  then  given  the  post-trial 
questionnaire  and  were  asked  to  complete  the  questionnaire,  and  return  it  to  the  class  instructor 
during  their  next  regularly  scheduled  class  period. 

2.2.8  Dependent  Measures 

Performance  Assessment 

The  major  performance  measures  were  derived  from  the  periodic  situation  assessment 
briefings,  or  sitreps.  Using  the  taped  oral  briefings  and  the  associated  written  materials,  a  SME 
evaluated  the  participants  on  four  aspects  of  their  performance:  1)  presence/absence  of  seven 
pieces  of  critical  information,  2)  overall  written  situation  assessment,  3)  overall  written  plus  verbal 
situation  assessment;  and,  4)  an  overall  situation  assessment  measure  created  by  looking 
at  the  ratings  of  both  the  information  pieces  and  the  situation  assessment. 

Information-Processing  Measures 

Of  the  76  message  sent,  1 1  were  determined  by  an  SME  to  be  mission  critical.  The 
remainder  were  considered  mission  nonrelevant.  The  number  of  mission  critical  and  mission 
nonrelevant  messages  opened  was  computed  as  one  assessment  of  information  filtering. 

Participants  were  required  to  rate  all  incoming  messages  that  they  opened  on  a  four-point  scale 
with  the  following  alternatives:  irrelevant,  non-critical,  moderately  critical,  highly  critical.  For 
analysis  purposes,  message  criticality  was  collapsed  into  two  categories:  noncritical  (those  rated 
irrelevant  or  non-critical  by  the  participant)  and  critical  (those  rated  moderately  or  highly  critical 
by  the  participant).  Messages  not  opened  by  participants  were  considered  as  having  been 
implicitly  rated  as  noncritical. 
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Participants  were  also  permitted  to  send  messages  to  different  nodes  in  the  organizational 
structure  to  ask  for  more  information  about  the  situation,  or  to  forward  messages  they  received  to  a 
node  that  they  felt  was  more  equipped  to  handle  the  information.  Outgoing  messages  could  be 
categorized  according  to  the  appropriateness  of  the  destination,  according  to  the  hierarchical  level 
(subordinate,  lateral,  or  super)  of  the  node  to  which  it  was  sent,  or  according  to  its  content. 

2.3  TRAINING  EVALUATION:  RESULTS 

In  this  subsection  we  discuss  the  results  of  the  IM  training  evaluation.  First  we  consider  the 

impact  of  organizational  knowledge  and  then  the  evaluation  of  the  IM  training  program.  We  look 
at  both  the  performance  and  the  processing  data. 

2.3.1.  Organizational  Knowledge 

We  hypothesized  that  subjects  in  the  deep  organizational  knowledge  condition  would  have 
higher  scores  on  the  organizational  knowledge  questionnaire  than  those  in  the  shallow  condition. 
Based  on  the  subjects’  responses  to  the  Organizational  Knowledge  Questionnaire  we  found  no 
difference  between  the  two  organizational  knowledge  conditions.  It  will  be  recalled  that  the 
organizational  knowledge  booklet  was  handed  out  ahead  of  time  to  be  read  by  the  participants 
before  the  start  of  the  first  training  session.  We  concluded  that  most  of  the  participants  in  the  deep 
organizational  knowledge  condition  looked  at  the  organizational  chart  and  associated  high  level 
materials,  but  did  not  devote  time  to  the  details  in  the  booklets  that  were  provided.  As  a  result, 
what  they  actually  looked  at  was  similar  to  what  the  participants  who  received  the  standard 
organizational  materials  saw,  and  this  similarity  was  reflected  in  the  similarity  between  the  two 
groups  on  their  responses  to  the  Organizational  Knowledge  Questionnaire. 

Because  there  were  no  differences  between  the  shallow  and  deep  organizational  knowledge 
conditions,  we  collapsed  this  variable  for  the  analysis  of  the  performance  and  processing  results, 

Training  Information  Management  28 

In  Distributed  Organizations 


ALPHATECH,  INC. 


and  focused  on  differences  between  the  group  that  received  the  IM  training  and  the  group  that 
received  the  control  training. 

2.3.2  Peformance  Assessment 

We  hypothesized  that  the  training  program  would  enhance  the  trained  subjects’  ability  to 
deal  effectively  with  large  amounts  of  information,  including  identifying  critical  pieces  of 
information.  We  expected  that  this  difference  would  be  reflected  in  more  complete  and  higher 
quality  sitreps  in  the  trained  group  than  in  the  control  groups.  To  assess  the  difference  between 
the  two  groups  we  examined  the  SME’s  ratings  of  the  subjects’  sitreps.  (The  rating  procedure  is 
described  in  subsection  2.2.8.) 

Contrary  to  our  expectation  we  found  no  difference  between  the  ratings  of  the  trained  and 
control  groups’  sitreps.  Figure  2-7  shows  the  number  of  critical  factors  out  of  seven  that  the 
subjects  correctly  identified  in  their  written  briefing  and  the  overall  rating  of  their  sitreps  based  on 
a  7-point  scale.  The  differences  between  the  two  groups  are  not  significant. 


Factors  Identified  Overall  Rating 

Performance  Variable 


Figure  2-7  Performance  Scores  for  Trained  and  Control  Subjects 
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One  of  the  striking  points  in  Figure  2-7  is  that  the  means  for  both  groups  are  quite  low.  On 
the  overall  rating,  a  score  of  1  means  the  subject  has  virtually  no  understanding  of  the  situation 
and  a  score  of  4  implies  only  limited  understanding  of  the  situation.  The  faculty  members  who 
supported  the  experiment  commented  to  us  that  the  scenario  was  quite  complex  and  perhaps  over 
the  heads  of  the  cadets.  When  they  were  doing  their  sitreps,  many  of  the  subjects  commented  that 
they  were  totally  lost.  The  low  mean  values  on  the  performance  variables  support  these 
assessments.  We  concluded  that  the  difficulty  of  the  scenario  and  the  problem  posed  in  the 
vignette  may  have  prevented  the  trained  subjects  from  using  what  they  had  learned  in  an  applied 
context. 

As  another  exploratory  approach  to  assess  the  impact  of  the  training,  we  identified  a  small 
subset  of  the  sample  who  evidenced  some  ability  to  deal  with  the  complex  scenario.  These  were 
subjects  who  had  an  overall  sitrep  rating  of  at  least  5  ( which  corresponded  to  a  qualitative  rating 
of  adequate  to  good).  There  were  seven  subjects  who  met  this  criterion,  four  trained  and  three 
control.  We  examined  their  scores  to  see  if  there  was  any  evidence  for  the  effectiveness  of  the 
training  in  this  group,  but  found  none.  The  mean  sitrep  rating  (5.3)  was  the  same  in  both  groups, 
and  the  mean  number  of  critical  factors  identified  was  virtually  the  same  in  both  groups  (3.8  for 
the  trained  and  3.7  for  the  control  subjects).  We  conclude  from  this  analysis  that  either  the 
training  was  not  strong  enough  for  participants  to  be  able  to  apply  it  to  a  problem  situation  or, 
even  for  the  somewhat  more  able  students,  the  problem  complexity  masked  any  training  effects. 

2.3.3  Information  Processing 

In  this  subsection  we  explore  differences  between  the  trained  and  control  subjects  in  their 
information  processing  behavior.  We  hypothesized  that  the  trained  subjects  would  have  a  clearer 
focus  on  their  information  requirements  and  information  sources,  and  this  would  make  them  more 
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effective  in  high  level  filtering  of  messages.  To  evaluate  this  hypothesis  we  compared  the  two 
groups  in  terms  of  which  messages  they  opened. 

We  looked  first  at  a  comparison  between  messages  received  from  another  node  in  the 
organization  versus  messages  received  from  external  sources,  such  as  media  or  other  civilian 
groups.  Figure  2-8  shows  the  proportion  of  messages  from  internal  and  external  sources  opened 
by  the  trained  and  the  control  subjects.  The  nature  of  the  external  sources  suggests  that  their 
messages  would  not  contain  critical  mission  information.  The  results  show  that  while  the  control 
subjects  opened  the  same  proportion  of  messages  from  within  and  outside  the  organization,  the 
trained  subjects  opened  a  lower  proportion  of  the  messages  from  external  sources.  The  result 
provides  some  confirmation  for  our  hypothesis  that  trained  subjects  were  more  effective  at  high 
level  information  filtering. 


Control  Trained 


Figure  2-8.  Control  and  Trained  Subjects’  High  Level  Filtering  by  Message  Source 
As  a  second  way  of  looking  at  high  level  filtering,  we  examined  filtering  based  on  the 
subject  of  the  message.  We  categorized  the  message  topics  into  two  groups:  those  that  were 
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clearly  not  relevant  to  the  subject’s  mission  and  those  that  were  potentially  relevant.  Figure  2-9 
shows  the  proportion  of  messages  of  each  type  opened  by  the  trained  and  the  control  subjects. 
The  figure  shows  while  both  groups  opened  a  smaller  percentage  of  nonrelevant  than  potentially 
relevant  messages,  the  trained  group  opened  more  potentially  relevant  and  fewer  nonrelevant 
messages  than  the  control  group.  This  result  provides  further  evidence  that  the  training  program 
enhanced  participants  high-level  filtering  ability. 


Figure  2-9.  Control  and  Trained  Subjects’  High  Level  Filtering  by  Message  Subject 


High  level  filtering  relies  only  upon  information  in  the  message  header.  When  the 
information  in  the  message  header  suggests  that  the  message  may  contain  information  relevant  to 
the  mission,  the  information  processor  must  open  and  examine  the  contents  of  the  message  to 
make  a  more  informed  decision  about  its  relevance.  To  help  individuals  assess  the  relevance  of 
messages  they  read,  the  Specify  step  in  the  MISSION  mnemonic  suggested  that  individuals  should 
specify  ahead  of  time  the  information  they  are  seeking,  and  the  Ignore  step  reminds  them  to  ignore 
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information  that  is  not  relevant  to  their  mission.  This  training  should  help  individuals  more 
accurately  assess  the  relevance  of  a  message  to  their  information  needs. 

Subjects  rated  messages  they  opened  on  a  4-point  scale,  where  1  was  not  relevant  at  all  and 
4  was  critical.  Figure  2-10  shows  the  control  and  trained  subjects  ratings  of  internally  and 
externally  generated  messages  that  they  opened.  Both  groups  rated  the  relevance  of  external 
messages  below  that  of  internal  messages.  The  difference  between  the  two  types  of  messages  was 
greater  for  the  trained  subjects  than  the  control  subjects,  but  the  interaction  was  not  significant. 


Figure  2-10.  Control  and  Trained  Subjects’  Message  Evaluation  by  Source  of  Message 


Figure  2.1 1  shows  a  similar  comparison  for  message  relevance.  Trained  subjects  rated 
potentially  relevant  messages  higher  and  nonrelevant  messages  lower  than  control  subjects,  but  the 
interaction  between  training  and  message  relevance  was  not  significant.  We  assessed  whether  the 
trained  subjects  were  more  accurate  in  differentiating  between  critical  and  noncritical 
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Figure  2-12.  Control  and  Trained  Subjects’  Message  Evaluation  by  Criticality  of  the  Message 
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messages  than  the  control  subjects.  All  subjects  rated  the  critical  messages  higher  than  the 
noncritical  ones,  but,  as  Figure  2-12  shows,  there  was  no  difference  between  the  two  groups. 

2.3.5  Message  Modality 

Half  the  subjects  in  each  experiment  condition  received  all  messages  in  a  written  format. 
The  other  half  received  approximately  40  percent  of  the  messges  in  spoken  format  and  the  other 
60  percent  in  written  format.  We  examined  the  information  processing  results  to  see  if  there  was 
any  difference  between  the  two  groups  on  this  factor. 

We  found  no  difference  between  the  two  groups  in  their  performance.  We  also  looked  at 
the  measure  of  workload  to  see  if  message  modality  impacted  the  subjects’  perceived  workload, 
but  found  no  difference  between  the  two  groups  on  this  measure  either. 

For  the  subjects  who  received  both  spoken  and  written  messages,  we  compared  their  rating 
of  the  importance  of  the  messages  to  see  if  message  modality  impacted  their  assessment  of  its 
importance,  but  found  that  the  mean  importance  ratings  were  about  the  same.  We  also  computed 
the  amount  of  time  subjects  kept  messages  open  to  see  if  the  message  modality  impacted 
processing  time.  On  average  the  subjects  kept  a  message  open  for  30  seconds,  and  the  mean  for 
both  written  and  verbal  messages  were  not  different  by  more  than  one  second. 

2.4  TRAINING  EVALUATION:  DISCUSSION 
2.4.1  Summary  of  Results 

Our  main  focus  in  the  analysis  of  the  data  from  the  training  program  evaluation  was  on 
exploring  differences  in  performance  and  message  processing  between  the  trained  and  the  control 
groups.  We  hypothesized  that  the  IM  training  program  would  make  participants  better  information 
managers,  better  able  to  identify,  evaluate,  and  comprehend  incoming  information,  better  able  to 
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recognize  information  gaps,  and  more  knowledgeable  about  where  and  when  to  seek  missing 
information. 

We  found  no  difference  between  the  two  groups  on  their  information  processing 
performance,  as  reflected  in  their  sitreps.  We  did  find  that  for  all  groups,  the  performance  ratings 
were  low,  with  few  subjects  even  reaching  the  level  of  satisfactory  performance. 

We  concluded  from  the  performance  results  that  the  scenario  materials  we  used  to  test  the 
effectiveness  of  the  training  program  were  above  the  experience  level  of  the  cadets  who 
participated  in  the  training  evaluation.  Given  this  situation,  it  is  not  possible  to  ascertain  whether 
the  observed  performance  resulted  because  training  did  not  have  an  impact  on  performance  or 
whether  our  ability  to  detect  any  effect  the  training  may  have  had  was  hampered  by  the  scenario 
we  used.  The  finding  that  even  among  the  higher-performing  subjects  there  was  no  discemable 
difference  between  the  trained  and  control  groups  suggests  that  any  effect  the  training  had  on 
performance  was  at  best  weak.  We  concluded  that  information  comprehension  and  integration  is 
an  area  that  required  strengthening  in  a  subsequent  version  of  the  training  program. 

We  did  find  that  the  training  had  an  impact  on  the  subjects’  information  processing 
behavior,  as  reflected  in  their  ability  to  filter  messages  before  opening  them.  We  found  that  the 
trained  subjects  were  more  effective  in  high  level  filtering  than  were  the  control  subjects.  They 
opened  fewer  messages  that  were  not  likely  to  be  relevant  to  their  mission.  For  messages  that 
subjects  opened,  the  impact  of  the  training  on  their  ability  to  identify  critical  messages  was  less 
clear.  Although  trained  subjects  differentiated  important  from  nonrelevant  messages  somewhat 
better  than  control  subjects,  the  differences  we  observed  were  not  statistically  reliable.  We 
concluded  that  the  training  program  seemed  to  be  oriented  in  the  right  direction,  but  was  not 
strong  enough  to  support  reliable  differences  in  subjects’  ability  to  discern  critical  information  in 
messages  they  open. 
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2.4.2  Implications  for  Training  Program  Enhancement 

Data  were  collected  from  a  number  of  sources  to  evaluate  the  training  program  as  it  was 
implemented  at  the  Military  Academy  and  to  aid  in  the  further  development  of  the  Information 
Management  training.  The  sources  included  data  collected  during  the  experiment;  comments 
collected  from  the  participants  in  the  training  program  and  the  faculty  members  who  were  involved 
in  the  experiment;  and  experimenter  observations.  These  sources  revealed  a  number  of  ways  in 
which  the  training  program  could  be  enriched,  modified,  and  extended.  Among  the  important 
conclusions  reached  are: 

•  Integrate  organizational  knowledge  training  into  IM  training  program 

In  the  preliminary  version  of  the  training  program  we  stressed  that  information  managers  need 
to  be  aware  of  the  organization  in  which  they  are  working,  including  the  kinds  of  information 
likely  to  be  available  at  different  nodes  in  the  organization.  However,  it  did  not  explicitly 
incorporate  organizational  knowledge  into  the  training  package.  Rather,  a  booklet  describing 
the  particular  organization  used  for  evaluating  the  training  program  was  distributed  to 
participants  to  be  read  before  they  arrived  for  training.  Data  collected  from  the  participants 
revealed  that  they  did  not  read  the  booklets  before  the  experiment,  and  since  the  contents  of 
this  booklet  were  not  specifically  related  to  the  training  program  and  were  not  explained 
verbally  in  any  way,  the  participants  did  not  achieve  the  level  of  organizational  understanding 
needed  for  effective  information  management.  They  were  not  motivated  and/or  able  to  apply 
the  information  described  in  the  organizational  knowledge  booklets.  Given  this  evidence,  we 
concluded  that  distribution  of  written  materials  is  not  a  dependable  way  to  convey 
organizational  knowledge.  Rather,  training  on  organizational  knowledge  should  be  directly 
integrated  into  the  IM  training  program.  This  training  would  emphasize  the  importance  of 
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organizational  knowledge  for  effective  information  management,  both  in  terms  of  information 
seeking  and  information  distribution. 

•  Explain  causes  and  consequences  of  information  overload  to  motivate  training 

Data  collected  from  the  participants  and  faculty  suggest  that  the  participants  lacked  the 
motivation  needed  to  stimulate  learning  in  a  training  situation.  Although  most  people  have 
experienced  information  overload,  fewer  are  aware  of  how  they  themselves  contribute  to 
situations  of  information  overload.  Likewise,  though  many  people  relate  situations  of 
information  overload  with  feelings  of  stress,  fewer  are  aware  that  overload  can  degrade 
cognitive  processing.  We  concluded  that  one  effective  way  to  motivate  the  training  program 
would  be  to  explain  how  individuals  can  unwittingly  contribute  to  information  overload,  for 
example  by  indiscriminately  forwarding  electronic  messages.  A  second  way  to  motivate  the 
training  is  to  explain  how  overload  impinges  on  different  aspects  of  cognitive  processing,  for 
example  by  narrowing  an  individual’s  focus  of  attention  and  by  decreasing  vigilance.  We  felt 
these  motivating  factors  would  be  especially  helpful  in  sensitizing  people  to  the  ways  in  which 
they  might  contribute  to  information  overload  and  its  impacts  on  their  cognitive  processing  and 
behavior. 

•  Put  a  strong  emphasis  on  behavioral  training 

In  addition  to  revealing  a  motivation  problem  with  our  participants,  our  data  also  suggested 
that  the  training  focused  too  little  on  actual  behaviors  that  should  be  used  for  proper 
information  management:  Participants  were  still  unsure  HOW  to  properly  manage 
information.  In  the  preliminary  version  of  the  training  program  we  used  the  elements  of  the 
MISSION  mnemonic  as  the  basis  for  training.  The  data  showed  some  indications  that  the 
program  was  effective  in  making  participants  better  at  high  level  filtering,  but  we  had  at  best 
weak  evidence  we  were  successful  in  improving  participants’  second  level,  or  content, 
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filtering.  We  concluded  that  the  training  approach  may  have  been  too  conceptual,  without 
showing  how  these  conceptual  ideas  could  be  behaviorally  implemented.  We  decided  that  the 
next  version  of  the  training  program  needed  to  incorporate  particular  information  processing 
skills  that  participants  can  use.  In  particular,  the  program  needed  to  stress  active  preparation 
of  information  requirements,  scanning  for  key  words  and  phrases  as  a  technique  for  evaluating 
the  relevance  of  incoming  information,  and  writing  of  succinct  messages  that  only  contain 
information  relevant  to  the  intended  receiver. 

•  Include  all  components  of  effective  training 

An  effective  training  program,  such  as  the  successful  Team  Adaptation  and  Coordination 
Training  (TACT)  Program  (Entin  and  Serfaty,  1999),  includes  four  components:  lecture, 
demonstration,  practice,  and  feedback.  In  the  preliminary  version  of  the  training  program  the 
demonstration  material  was  packaged  as  a  written  document,  and  appeared  to  be  another  facet 
of  the  lecture  component.  We  concluded  that  an  effective  demonstration  component  needs  to 
be  more  dynamic,  and  to  rely  on  examples  acted  out  in  a  specific  context,  for  example  by 
incorporating  video  clips  that  portray  examples  of  inappropriate  and  appropriate  information 
processing  behaviors.  Furthermore,  due  to  a  limit  on  the  amount  of  training  time  that  was 
available,  we  did  not  include  practice  and  feedback  components  into  the  preliminary  version. 
From  the  data  collected,  it  was  clear  that  the  lack  of  opportunity  for  participants  to  practice  the 
concepts  and  behaviors  that  were  covered  in  the  lecture  and  demonstration  components  limited 
their  ability  to  internalize  the  material  that  was  covered.  We  concluded  that  even  with 
improved  lecture  and  demonstration  components,  the  effectiveness  of  training  is  weakened 
without  the  opportunity  for  participants  to  apply  the  concepts  and  receive  feedback  on  whether 
they  are  applying  them  appropriately. 
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•  Enhance  and  ensure  sensitivity  of  training  evaluation 

The  finding  that  the  scenario  and  task  we  used  to  evaluate  the  training  program  was  too 
complex  for  the  participants  in  the  training  program  weakened  our  ability  to  draw  conclusions 
about  the  effectiveness  of  the  training.  It  emphasized  the  need  to  insure  that  the  scenario  and 
task  are  calibrated  to  the  trainees’  level  of  experience.  It  also  alerted  us  to  the  need  to  insure 
that  the  examples  used  in  the  training  program  are  realistic  and  at  a  level  appropriate  for  the 
individuals  who  are  being  trained.  Furthermore,  in  the  course  of  analyzing  the  data  from  the 
training  evaluation,  we  realized  that  the  sources  and  subjects  of  the  messages  used  were  too 
limited,  and  that  we  needed  to  have  a  greater  range  of  sources  and  subjects  both  in  the 
messages  used  for  the  training  component  and  for  those  used  in  the  evaluation  of  the  training. 
We  also  concluded  that  the  information  about  time  messages  were  generated  was  not  salient, 
and  in  so  far  as  time  is  a  factor  that  is  considered  in  information  filtering,  the  message  time 
needed  to  be  made  clearer. 

•  IM  training  should  build  on  the  training  already  experienced  by  the  military  personnel 

The  preliminary  version  of  the  training  program  was  a  stand-alone-training  program;  although 
the  examples  used  were  military  in  nature,  no  consideration  was  given  to  the  decision  making 
training  delivered  to  military  personnel.  This  approach  resulted  in  training  that  was  at  times 
redundant  to,  and  at  times  conflicting  with,  the  training  the  personnel  already  receive.  Because 
the  participants  were  cadets,  whose  military  training  was  just  beginning,  any  inconsistencies 
that  might  have  existed  were  not  evident.  However,  to  make  the  training  more  effective  for 
experienced  officers,  and  to  capitalize  on  the  training  already  received  by  the  military 
personnel,  we  consulted  with  an  SME  who  suggested  that  the  training  should  be  revised  in 
such  a  way  that  it  would  integrate  more  seamlessly  with  the  Military  Decision  Making 
Procedures  (MDMP)  that  officers  are  taught. 
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Based  on  these  conclusions,  we  worked  in  conjunction  with  an  SME  to  develop  and 
implemented  another  version  of  the  IM  training  program.  We  applied  and  evaluated  this  improved 
version  with  participants  from  the  Battle  Command  Battle  Laboratory  at  Fort  Leavenworth,  KS. 
We  describe  the  improved  program  and  the  evaluation  that  was  conducted  in  the  next  chapter. 
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CHAPTER  3 

INFORMATION  MANAGEMENT  TRAINING  PROGRAM:  DEVELOPMENT  AND 

EVALUATION  OF  THE  TEST  VERSION 

In  this  chapter  we  discuss  the  enhancement  and  restructuring  of  the  preliminary 
information  management  (IM)training  program  and  the  evaluation  experiment  that  we  conducted 
to  assess  the  effectiveness  of  the  revised  training  program.  We  refer  to  this  upgraded  version  of 
the  training  program  as  the  Test  Version. 

3.1  DEVELOPMENT  OF  THE  TEST  VERSION 

3.1.1  Modifications  and  Enhancements  to  Preliminary  Version 

Focus  Restructuring 

The  preliminary  version  of  the  training  program  featured  a  mnemonic,  “MISSION”  to 
convey  the  substantive  points  of  the  training,  particularly  the  need  to  identify  and  focus  on  critical 
elements  of  information.  Emphasis  was  placed  on  preparation,  including  information 
requirements,  information  sources,  and  time  constraints  for  obtaining  information.  Less  emphasis 
was  placed  on  how  to  apply  this  knowledge  to  specific  information  needs.  In  the  Test  Version  we 
used  a  different  approach.  Although  the  “MISSION”  concept  seemed  appropriate  for  the  target 
audience  who  received  the  preliminary  version  of  the  training  program,  our  subject  matter  expert 
(SME)  advised  us  that  the  word  “mission”  has  very  specific  connotations  for  military  officers,  and 
the  way  in  which  we  were  using  the  term  might  seem  confusing  or  inappropriate  to  them.  He  also 
advised  us  that  the  approach  we  used  needed  to  be  consistent  with  military  training  that  officers 
have  received,  and  that  we  needed  to  show  where  it  fit  in  terms  of  the  traditional  Military  Decision 
Making  Process  (MDMP)  that  officers  learn. 

Another  reason  why  we  moved  away  from  the  “MISSION”  concept  is  that  we  concluded 
that  the  training  needed  to  focus  on  specific  information  processing  behaviors  as  well  as  stressing 

Training  Information  Management  42 

In  Distributed  Organizations 


ALPHATECH,  INC. 


the  importance  of  determining  information  requirements.  That  is,  it  needed  to  address  techniques 
that  can  be  applied  to  deal  effectively  with  incoming  and  outgoing  information  in  a  dynamic 
information  processing  situation. 

Because  military  officers  might  assume  that  the  military  had  already  given  them  all  the  IM 
training  that  they  would  need,  we  wanted  to  highlight  the  need  for  additional  training  to  motivate 
the  subjects  to  attend  to  the  training.  We  did  this  by  framing  our  discussion  in  the  context  of  how 
the  MDMP  is  an  excellent  process  to  follow  when  faced  with  decision  making  situations,  and  how 
this  new  training  supplements  the  MDMP  by  giving  trainees  specific  behaviors  that  they  can  use  to 
make  the  MDMP  work.  To  further  motivate  trainees,  we  discussed  the  adverse  impacts  of 
information  overload  and  the  degradation  in  cognitive  processing  that  can  result  from  information 
overload. 

Another  conclusion  from  the  evaluation  of  the  preliminary  version  was  that  organizational 
knowledge  needed  to  be  folded  in  as  a  component  of  the  IM  training.  Giving  participants  long 
manuals  on  the  organizational  structure  and  channels  of  communication  to  read  on  their  own 
proved  to  be  an  ineffective  way  to  provide  organizational  knowledge.  For  the  Test  Version  we 
incorporated  training  about  the  importance  of  knowing  the  organization  for  effective  information 
management  directly  into  the  IM  training  program. 

Training  Components 

The  preliminary  version  of  the  training  program  was  comprised  of  two  components: 
lecture  and  demonstration.  The  demonstration  portion  consisted  of  a  discussion  based  on  written 
examples  of  ineffective  and  effective  information  management.  In  administering  the  program,  we 
noted  that  the  written  examples,  though  substantively  adequate,  were  rather  ‘flat’  when  they  were 
presented  as  text.  A  more  effective  technique,  we  concluded,  would  be  to  make  these  examples 
more  salient  and  more  credible  by  having  them  acted  out.  In  the  Test  Version  of  the  training 
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program,  we  created  a  demonstration  videotape,  in  which  ineffective  and  effective  IM  techniques 
were  exemplified.  Entin  and  Serfaty,  (1999)  used  this  approach  successfully  in  a  training  program 
they  developed. 

Due  to  limitations  on  the  amount  of  time  available  for  training,  we  did  not  incorporate 
the  practice  and  feedback  components  of  training  into  the  preliminary  version.  The  lack  of 
opportunity  for  the  trainees  to  apply  the  concepts  discussed  in  the  lecture  and  demonstration 
portions  of  the  training  emphasized  the  importance  of  the  practice  and  feedback  components.  For 
the  Test  Version,  we  developed  a  specific  training  scenario  that  allowed  subjects  to  practice  the 
concepts  that  had  been  covered  in  the  lecture  and  demonstration  phases  of  the  training.  Feedback 
on  performance  could  go  hand  in  hand  with  practice. 

3.1.2  Description  of  the  Test  Version  of  the  Training  Program 

The  Training  Program  was  comprised  of  the  four  components  needed  for  effective 
training:  lecture,  demonstration,  practice,  and  feedback.  The  lecture  portion  was  conducted  by  the 
trainer,  and  was  organized  around  a  set  of  slides  that  set  the  context  of  the  training,  provided 
motivation  for  it,  and  discussed  the  training  concepts  and  techniques  that  we  had  developed.  We 
used  the  initial  portion  of  the  lecture  component  of  the  training  program  to  convey  how  the  IM 
training  we  provided  fit  in  with  previous  military  training  and  to  motivate  the  need  for  effective  IM 
skills. 

We  wanted  to  make  it  clear  at  the  outset  of  the  training  that  it  builds  upon  and  supplements, 

and  is  not  a  repetition  of  or  replacement  for,  the  information  processing  training 

that  Army  officers  have  already  received.  The  trainer  used  the  diagram  shown  in  Figure  3-1  to 

provide  an  overview  of  the  training  program  and  show  how  it  fits  into  the  Army’s  Military 

Decision-Making  Process  (MDMP).  Figure  3.1  also  provides  an  overview  of  the  key  information 

processing  techniques  presented  in  the  training  program. 
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The  Military  Decision-Making  Process  (MDMP) 

Information  Management  Training 

Information  management  starts  in 

Receipt  of  Mission 

Mission  Analysis 

Mission  Analysis 

Prep:  Prepare-  What  information  is  needed  (key 
words  and  phrases);  where  this  information  might 
come  from;  differentiate  information  that  would  fit 
in  the  different  levels  of  importance. 

COA  Development 

Filter:  Initial  filter-  filter  on  source,  subject,  and 

COA  analysis 

Scan:  Open  and  scan  those  messages  that  pass  the 

GOA  Comparison 

Sfirst  filter;  look  for  the  key  words  and  phrases 
i; ^established  in  the  Prep  phase. 

COA  Approval 

\  .:••• 

Read:  Read  carefully  those  messages  that  pass  the 
first  two  filters  (Filter  and  Scan) 

Orders  Production 

Act:  Take  appropriate  action.  Based  on  findings  in 
Prep  phase,  determine  what  level  of  importance  the 

Rehearsal 

message  is.  If  the  message  contains  information 
needed  by  other  nodes,  quickly  write  anew 

Execution  and  Assessment 

message  that  details  only  the  important  information 
and  send  it 

Figure  3-1 .  Relationship  of  Information  Management  Training  to  MDMP  and  Key  Information 

Processing  Techniques 


Once  we  established  the  context  for  the  training  program,  the  trainer  turned  to  a  discussion 
of  what  information  overload  is,  why  it  occurs,  and  what  are  the  cognitive  and  psychological 
results  of  information  overload.  With  this  contextual  and  motivational  basis  established,  the 
lecture  went  on  to  discuss  the  techniques  portrayed  in  Figure  3-1  (prepare,  filter,  scan,  read,  act)  in 
detail,  using  several  illustrative  scenarios.  Examples  of  effective  and  ineffective  applications  of 
these  techniques  were  provided  in  the  context  of  the  illustrative  scenarios. 

The  demonstration  portion  of  the  training  program  centered  around  videotaped  examples 
illustrating  less  and  more  effective  IM  techniques.  The  examples  presented  were  based  upon  a 
scenario  in  which  an  officer  needed  to  monitor  email  traffic  to  determine  whether  and  how  units 
could  get  around  a  wadi.  The  videotape  examples  showed  someone  handling  information  in  a  less 
than  optimal  way,  and  then  showed  the  same  person  doing  a  more  effective  job.  After  each 
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segment  the  trainer  stopped  the  tape  and  discussed  the  contents  with  the  trainees.  Examples  were 
provided  for  each  of  the  techniques  discussed  in  the  lecture  portion,  starting  with  identifying 
information  requirements,  high  level  filtering,  information  scanning,  and  information  distribution. 

In  the  practice  and  feedback  portion  of  the  training,  the  trainee  was  given  a  role  as  an 
assistant  battle  captain  whose  task  it  was  to  monitor  email  traffic  to  determine  why  an  armored 
column  had  stopped  along  a  road.  Before  the  vignette  was  described  to  them,  participants  were 
given  a  Preparation  Sheet  on  which  they  could  make  notes  about  information  requirements  as  the 
vignette  was  described  and  as  the  scenario  unfolded.  A  trainer  sat  with  the  trainee  as  he  or  she 
processed  the  email  messages,  answering  questions  the  trainees  had  about  the  contents  of  the 
messages,  and  providing  some  feedback  on  their  message  processing  actions.  The  practice  also 
allowed  subjects  to  become  acquainted  with  the  features  of  the  simulator  we  were  using,  and  the 
requirement  for  rating  both  incoming  and  outgoing  messages. 

3.1.3  Description  of  Control  Training 

In  order  to  assess  the  effectiveness  of  the  training  program,  we  needed  a  control  group 
against  which  to  compare  the  trained  group  on  an  IM  exercise.  The  control  group  received 
training  on  the  Objective  Force.  It  was  explained  that  the  scenario  they  would  be  participating  in 
was  set  in  the  future  (2008),  and  since  allied  forces  were  joint,  some  background  on  the  vision  for 
the  Objective  Force  would  set  the  stage  for  the  exercise. 

The  lecture  portion  for  the  control  portion  was  somewhat  longer  than  the  IM  training 

lecture  component,  and  was  followed  by  a  discussion  between  the  trainer  and  trainees.  The 

control  training  did  not  contain  a  demonstration  segment.  Following  the  lecture  and  discussion, 

the  control  subjects  went  into  the  practice  and  feedback  portion  of  the  training.  The  procedure 

was  similar  in  form  to  that  used  for  the  IM  training  subjects,  except  that  feedback  did  not  include 

guidance  on  the  manner  in  which  the  subjects  were  processing  the  email  messages.  The  control 
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trainers  did  answer  factual  questions  subjects  had  about  the  scenario,  and  any  questions  they  had 
about  use  of  the  email  application. 

3.2  TRAINING  EVALUATION:  METHOD 
3.2.1  Scenario  Materials 

We  used  the  same  scenario  materials  that  were  used  for  Experiment  1  and  for  the 
assessment  of  the  preliminary  version  of  the  IM  training  program.  They  were  modified  by  an  SME 
to  make  them  more  consistent  with  Army  doctrine  and  practice  and  to  make  the  force  organization 
clearer.  The  scenario  portrays  a  Joint  mission  situation  set  in  2008.  The  basic  situation  is  that 
Country  A’s  troops  have  invaded  Country  B,  and  a  Joint  Coalition  Force,  Task  Force  Agile,  led  by 
the  United  States,  has  been  authorized  to  restore  Country  B’s  borders.  After  12  days  of  attacks 
against  Country  A’s  forces,  intelligence  estimates  indicate  that  its  air,  missile  and  naval  forces  in 
Country  B  have  been  largely  destroyed  and  communications  capabilities  crippled.  Based  on  these 
estimates,  the  commander  of  Task  Force  Agile  has  just  determined  that  Phase  I  of  the  campaign 
has  been  successfully  completed,  and  has  ordered  execution  of  Phase  II.  The  two  vignettes  that 
were  used  for  training  practice  and  for  training  evaluation  stemmed  from  this  basic  scenario.  The 
following  sets  of  materials  were  used  to  portray  the  scenario: 

Background  Situation  (Road  to  War). 

This  document  provided  a  description  of  the  background  situation,  including  how  and  why 
the  Country  A’s  forces  were  in  Country  B,  the  composition  and  disposition  of  Task  Force  Agile, 
and  the  coalition  plans  for  routing  the  Country  A’s  forces.  It  describes  the  makeup  of  the  coalition 
forces,  and  outlines  the  activities  that  have  occurred  under  Phase  I. 
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Oplan 

The  Oplan  consists  of  the  mission,  commander’s  intent,  the  concept  of  operations  from 
Phase  I  to  Phase  V,  and  a  more  detailed  plan  for  Phase  II,  in  which  the  vignettes  are  set.  It  also 
includes  a  partial  intelligence  annex  and  operations  graphics. 

Intelligence  Annex 

This  Annex  describes  the  enemy  laydown.  It  was  not  deemed  highly  relevant  for  this 
mission  but  was  included  for  the  sake  of  realism  and  completeness. 

Situation  Update 

This  one-paragraph  document  brings  the  situation  up  to  the  current  time  in  which  the  two 
vignettes  are  set.  It  states  that  Phase  I  of  the  operation  has  been  successfully  completed,  and  the 
Commander  of  Task  Force  Agile  has  authorized  the  start  of  Phase  II. 

Training  and  Data  Collection  Vignettes 

Working  with  a  SME,  we  developed  two  vignettes,  one  to  be  used  for  the  practice 
component  of  the  training  and  the  other  for  the  training  evaluation.  Both  spin  off  from  the  basic 
scenario.  Both  vignettes  were  designed  to  be  interesting  to  and  at  a  suitable  level  of  difficulty 
for  mid-career  Army  officers.  For  both  vignettes,  the  subject  played  the  role  of  an  assistant 
operations  officer  in  one  of  the  forces  comprising  Task  Force  Agile.  The  subject’s  task  was  to 
monitor  message  traffic  and  assess  an  evolving  situation.  For  each  vignette  we  provided  a 
communications  chart  showing  the  primary  and  secondary  communications  linkages  in  the 
organizational  structure. 

The  vignette  used  for  training  was  adapted  from  one  we  had  used  previously  to  make  it 
more  Army-oriented.  In  the  training  vignette  the  participant,  playing  the  role  of  battle  captain,  is 
charged  with  the  task  of  determining  why  an  enemy  armored  column  has  stopped  in  an  area  in 
which  Blue  intended  to  land  troops  by  helicopter  as  part  of  a  forced  insertion.  The  battle  captain  is 
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told  that  at  the  “eleventh  hour”  a  humint  group  on  the  ground  reports  that  an  armored  column 
of  tanks  and  other  armored  vehicles  has  halted  in  the  intended  drop  zone.  The  battle  captain  is 
charged  with  the  task  of  monitoring  the  information  traffic  and  determining  the  intent  of  the  enemy 
armored  column.  In  particular,  have  they  compromised  Blue’s  plans  and  intend  to  defend  the 
area?  Are  they  reinforcing  other  units  in  the  area?  Have  they  stopped  for  some  other  reason? 

In  the  data  collection  scenario  the  participant,  playing  the  role  of  battle  captain,  is  asked  to 
determine  if  an  enemy  mechanized  division  is  heading  toward  or  away  from  the  area  in  which  Blue 
plans  a  forced  insertion.  The  battle  captain  is  told  movement  has  been  detected  in  an  enemy  division. 
The  task  is  to  monitor  the  information  traffic  and  determine  the  direction  in  which  the  mech  force  is 
moving.  Blue’s  intel  officer  provides  three  plausible  enemy  intentions:  the  division  is  moving  to  the 
north  in  response  to  Blue’s  deception  to  convince  the  enemy  that  the  invasion  will  occur  in  the  north. 
This  is  considered  most  likely  and  favorable  for  Blue  in  that  it  indicates  the  enemy  has  bought  into  the 
deception  and  is  moving  its  division  too  far  away  to  oppose  Blue’s  forced  insertion.  A  second 
possibility  is  that  the  enemy  division  is  moving  northeast  to  a  sea  port.  This  is  also  plausible  and  far 
enough  away  from  the  planned  insertion  point  to  not  foul  Blue’s  plan.  The  third  alternative,  the  most 
dangerous  for  Blue,  holds  that  the  enemy  division  is  moving  south  into  the  general  area  of  the  forced 
insertion.  This  would  mean  Blue  would  have  to  suspend  its  current  plan  and  turn  to  an  alternate. 
Based  on  the  email  traffic  contained  in  the  scenario,  the  battle  captain  must  determine  which  enemy 
alternative  is  most  likely  and  report  back  to  his  commander  in  his  sitrep. 

Maps 

We  provided  each  subject  with  an  ONC  (1:1 ,000,000)  map  opened  to  the  area  in  which 
the  scenario  is  set.  Overlaid  on  the  map  was  a  hand-drawn  overlay  which  portrayed  relevant 
locations  of  Task  Force  Agile  units  and  a  sketch  of  the  general  plan  for  the  Phase  II  operation..  For 

Training  Information  Management  49 

In  Distributed  Organizations 


ALPHATECH,  INC 


each  of  the  vignettes,  we  provided  a  hand-drawn  overlay  that  showed  the  force  distribution 
specific  to  that  vignette. 

Messages 

For  each  vignette,  the  dynamic  scenario  was  described  though  a  set  of  messages  that 
portray  an  evolving  situation  stemming  from  the  vignette  descriptions.  Each  message  was 
categorized  by  the  SME  who  developed  the  vignette  in  terms  of  its  importance  for  the  mission 
described  in  the  vignette.  The  importance  level  of  the  email  messages  was  determined  by  the  SME 
who  developed  the  data  collection  vignette.  Critical  messages  were  those  that  were  necessary  for 
understanding  the  evolving  situation.  They  pertained  directly  to  the  commander’s  critical 
information  requirements  (CCIRs).  Relevant  messages  supported  understanding  of  the  situation, 
but  any  single  message  did  not,  in  and  of  itself,  convey  information  that  was  crucial  for 
understanding  the  situation.  Nonrelevant  messages,  although  they  might  be  important  in  some 
other  context,  were  not  related  to  the  subject’s  mission.  Appendix  B  contains  examples  of  the 
three  types  of  messages. 

Data  Collection  Vignette  (80  Messages) 


Figure  3-2  Distribution  of  Importance  Ratings  of  Messages  in  Data  Collection  Scenario 
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In  developing  the  message  stream,  the  goal  was  include  more  and  less  relevant  messages  in 
a  ratio  similar  to  what  might  occur  in  a  real  world  situation.  Figure  3-2  shows  the  distribution  of 
importance  ratings  for  the  80  messages  in  the  data  collection  scenario. 

3.2.2  Scenario  Generator 

The  simulator  used  for  the  training  program  evaluation  was  a  tailored  version  of  a  standard 
email  application,  similar  to  the  one  used  at  the  Military  Academy  in  the  evaluation  of  the 
preliminary  version  of  the  training  program.  It  allows  an  experimenter  to  set  up  a  file  of  pre¬ 
planned  messages,  and  send  the  messages  at  specified  times  after  start  of  scenario  execution  from 
a  ‘controller’  machine  to  a  ‘subject’  machine.  It  also  allows  both  the  controller  and  the  subject 
machines  to  reply  to  messages,  forward  messages,  and  generate  new  messages  at  any  time  during 
the  execution  of  the  scenario.  It  logs  both  the  subject  and  the  controller  message  processing 
actions  in  a  message  file,  thereby  providing  a  record  of  which  messages  were  opened,  the  times 
they  were  opened  and  closed,  and  the  ratings  of  the  messages.  It  requires  the  person  at  the  subject 
machine  to  rate  the  importance  of  all  messages  that  are  read  and  all  messages  that  are  sent. 

The  following  subsections  describe  the  system  and  hardware  requirements  for  running  the 
scenario  generator,  the  modifications  we  made  to  the  initial  version  used  in  the  evaluation  at  West 
Point,  and  a  description  of  the  interface  for  this  upgraded  version. 

System  Requirements 

The  software  is  written  in  Java™,  using  the  Java  Developers  Kit  1 .2.x  (JDK™).  It  requires 
JavaBeans™  Activation  Framework  1.0,  JavaMail™  1.1.3,  and  POP3  1.1.  These  are  freely 
available  from  the  Sun  website,  http  ://iava.  sun.com/products.  The  program  runs  on  a  PC  running 
Windows  NT™.  It  requires  about  500kb  of  disk  space  to  install,  and  several  additional  kilobytes 
of  disk  space  to  write  the  log  files  each  time  it  is  run.  It  requires  access  to  a  machine  that  can 
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function  as  a  POP3  and  an  SMTP  server.  Two  accounts  are  required  on  the  server.  An  external 
program  (such  as  Netscape  or  Eudora)  must  be  used  to  remove  messages  from  the  server. 

The  software  runs  on  a  pair  of  machines.  One  machine  functions  as  the  controller,  and 
sends  a  stream  of  preplanned  messages  at  specified  times.  The  other  is  the  subject  machine,  which 
receives  the  messages. 

Modification  of  Message  Handling  program 

Several  modifications  and  enhancements  were  made  to  the  scenario  generator  for  the 
Test  Version.  Key  modifications  that  were  made  included: 

1)  Augmenting  the  log  file  of  messages  received  by  the  subject  to  include  ground  truth  priority 
and  message  generator  (to  differentiate  preplanned  messages  from  dynamically  generated 
messages  sent  by  the  controller),  and  modifying  the  format  of  the  file  to  make  it  more  suitable 
for  data  analysis. 

2)  Changing  the  format  of  displayed  message  header  to  make  it  more  legible  and  adding  a  “to” 
field  to  specify  all  recipients  of  the  message. 

3)  Augmenting  the  log  file  of  messages  sent  by  the  subject  to  include  a  new  field  indicating  the 
type  of  message  (reply,  forward,  forward  with  edits,  new),  and  tracking  which  source  message 
is  being  replied  to  or  forwarded. 

4)  Creating  a  log  file  of  messages  sent  by  the  controller,  similar  to  the  log  file  of  messages  sent  by 
the  subject. 

5)  Modifying  the  message  ID  to  allow  tracking  of  individuals  messages  when  a  “conversation”  of 
replies  takes  place. 

6)  Developing  scripts  to  simplify  running  of  the  program  and  data  retrieval. 
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Scenario  Generator  -  email  program 

The  simulator  delivered  new  email  messages  every  30  seconds.  Each  preplanned  message 
was  sent  out  at  a  predetermined  time.  With  a  total  of  80  preplanned  messages  and  a  30  minute 
scenario  time,  messages  arrived  at  the  rate  of  approximately  2.7  per  minute.  Given  that  messages 
were  delivered  in  30  second  intervals,  typically  one  or  two  messages  arrived  in  each  30  second 
interval. 

Figure  3-3  shows  a  sample  email  message  window.  Messages  in  black  have  not  been 
opened;  grayed  out  messages  have  already  been  opened.  Figure  3.4  shows  an  example  of  an  open 
message  with  the  importance  rating  menu  pulled  down.  In  the  figure,  the  subject  has  not  yet 
selected  the  importance  rating  for  the  open  message. 

Subjects  were  able  to  reply  to,  forward,  and  generate  new  messages.  These  options  are 
shown  at  the  top  of  Figure  3-3.  If  a  subject  chooses  reply,  the  original  message  appears  in  the  box, 
and  the  subject  can  start  typing  a  reply,  with  or  without  deleting  the  original  message.  If  a  subject 
chooses  to  forward  a  message,  the  original  message  appears  in  the  box  and  the  subject  can  send  it 
as  is,  or  modify  and/or  add  to  the  original  message.  If  the  subject  chooses  new,  an  empty  message 
box  appears,  and  subject  can  start  typing  a  message. 

The  subject  had  to  specify  the  recipient  of  the  messages,  and  a  scrollable,  pull  down  menu, 
shown  in  Figure  3-5,  was  available  for  that  purpose.  The  last  selection  in  the  scrollable  menu  was 
‘all’.  If  the  subject  chose  that  alternative,  the  message  would  be  sent  to  all  nodes  in  the 
organization.  Subjects  were  also  required  to  rate  the  importance  of  any  messages  they  sent  out. 
The  menu  for  importance  rating  of  outgoing  messages  is  below  the  Subject  Line.  It  is  also  a  pull¬ 
down  menu.  (In  Figure  3-5  it  is  obscured  by  the  list  of  message  recipients.) 
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Figure  3-3.  Sample  e-mail  Message  Window  Message  Generation  Options  on  Top 
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Figure  3-4.  Sample  email  Message  with  Important  Rating  Menu 
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Figure  3-5.  Scrollable  List  of  Recipients  for  Subject-Generated  Messages 


All  subject-generated  messages  were  received  at  the  controller’s  station,  regardless  of 
the  node  to  which  they  were  sent.  If  the  controller  chose  to  do  so,  he  or  she  could  reply  to  the 
subject’s  messages,  for  example  to  answer  a  question  or  to  confirm  a  requested  action.  Controller¬ 
generated  messages  were  sent  back  to  the  subject,  and  appeared  in  addition  to  the  preplanned 
messages.  The  rate  of  message  delivery  was  increased  somewhat  if  the  controller  sent  out 
messages.  To  retain  uniformity  in  the  substance  of  the  message  stream,  the  controller-generated 
messages  were  short  and  noncommittal.  For  example,  if  the  subject-generated  message  was  an 
action  request,  the  controller  might  reply  “OK”,  or  “will  do”.  If  the  subject’s  message  contained  a 
question,  controller  might  reply  something  like  “will  check  that  out”,  or  “not  known  at  this  time”, 
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as  appropriate.  The  goal  was  to  give  subjects  some  response  so  that  they  knew  their  messages 
were  being  read,  but  not  add  any  substantive  information. 

3.2.3  Instruments 

Five  instruments  were  used  during  the  training  evaluation,  all  of  which  were  completed  by 
the  subjects.  (See  Appendix  B  for  copies  of  these  instruments.) 

The  Background  Questionnaire  solicited  information  from  the  subjects  about  their 
military  background,  including  their  years  of  service,  rank,  and  specialty.  It  also  requested 
information  about  their  recent  military  schooling  and  exercises  in  which  they  had  participated. 

The  Background  Questionnaire  was  administered  at  the  beginning  of  the  experiment  session. 

The  Post-Training  Questionnaire  was  administered  to  assess  the  subjects’  understanding 
of  the  material  covered  in  the  training  program.  It  contained  five  short-answer  questions,  of  which 
three  were  relevant  to  the  training  experience,  and  two  were  filler  questions,  one  of  which 
incorporated  topics  discussed  in  the  control  ‘training’.  This  questionnaire  was  administered  after 
the  practice  and  feedback  portion  of  the  training. 

The  Briefing  Organization  Form  helped  subjects  prepare  their  interim  and  final  sitreps 
It  included  four  questions  about  the  information  they  would  like  to  have,  where  they  could  obtain 
it,  their  current  hypothesis  about  the  situation,  and  how  they  thought  it  would  evolve  that  subjects 
could  use  to  organize  their  thoughts.  Subjects  were  required  to  complete  this  form  in  preparation 
for  the  interim  briefing.  They  were  encouraged,  but  not  required,  to  fill  out  the  Briefing 
Organization  Form  to  prior  to  giving  their  oral  sitrep. 

The  Briefing  Preparation  Sheet  was  a  single  page  of  lined  paper  that  could  be  used  by 

subjects  to  take  notes  during  the  scenario  preparation  or  message  handling  phase  of  the  experiment.  It 

was  given  to  subjects  prior  to  the  start  of  the  description  of  the  training  and  data  collection  vignettes. 

Subjects  were  told  they  could  use  this  paper  to  jot  down  any  notes  they  wanted  about  the  scenario. 
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The  Post  Experiment  Questionnaire  was  a  self-report  measure  that  probed  subjects’  views 
about  the  comprehensibility  of  the  scenario,  their  actions  during  message  processing,  and  their 
overall  evaluation  of  the  training  program.  It  was  administered  at  the  end  of  the  experiment 
session,  just  prior  to  the  subject  debriefing. 

3.2.4  Procedure 

Three  experimenters  participated  in  the  training  evaluation.  One  experimenter  served 
as  the  trainer  for  both  the  trained  and  control  groups.  A  second  experimenter  presented  all  the 
scenario  materials.  These  two  experimenters  also  served  as  the  controllers  during  the  conduct  of 
the  training  and  data  collection  scenarios.  A  third  experimenter  conducted  the  training  on  the 
messaging  system,  and  served  as  an  experiment  coordinator. 

Figure  3-6  provides  a  high-level  overview  of  the  training  evaluation  design  and  procedure. 
Each  session  lasted  approximately  three  hours,  and  was  designed  either  as  a  ‘training’  or  a 
‘control’  session.  Two  subjects  participated  in  each  session.  At  the  start  of  each  session,  an 
experimenter  provided  an  introduction  and  overview  of  the  session  format.  Subjects  then 
completed  the  Background  Questionnaire. 


Initial  Training 

Experiment  Group 
information  Mgmt  Lecture  and  Demo 

Control  Group 
Future  Army  Lecture/Discussion 


Scenario  Presentation 
Road  to  War 
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1  T - 

Hands-On  Training 

Data  Collection 

End  of  Session 

E-mail  system  Overview 
Practice  Vignette  Presentation 
Scenario  Practice  and  Feedback 

►  Message  Processing 
Oral  Sitreps 

Post-Experiment  Questionnaire 
Debriefing 

Figure  3-6.  Overview  of  Experiment  Procedure 
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The  initial  training  was  comprised  of  the  lecture  and  demonstration  (or  discussion  for  the 
control  group),  and  lasted  approximately  40  minutes.  The  two  subjects  were  trained  together  for  this 
portion  of  the  training  program.  Subjects  in  the  training  (i.e.,  experiment)  group  received  the  IM 
training  materials.  Subjects  in  the  control  group  received  the  materials  on  the  Army  of  the  future. 

Following  the  lecture  and  demonstration  training,  subjects  were  briefed  on  the  scenario  that 
would  be  used  for  both  the  practice  and  data  collection  portions  of  the  experiment.  The  briefing 
included  an  overview  of  the  background  situation  (The  Road  to  War)  and  the  commander’s  Oplan 
for  Phase  II  of  the  operation,  the  phase  in  which  both  the  practice  and  data  collection  vignettes  were 
placed.  Following  that,  subjects  received  a  short  introduction  to  the  modified  email  system  that  served 
as  the  simulator.  The  major  focus  of  this  introduction  was  to  apprise  subjects  of  the  message  rating 
requirement.  Then  subjects  read  the  scenario  update,  a  short  paragraph  stating  that  Phase  I  of  the 
operation  had  been  successfully  completed.  One  of  the  experimenters  then  briefed  the  subjects  on  the 
training  vignette,  explained  their  role  in  the  organization,  and  posed  the  problem  situation  they  were 
required  to  assess.  At  the  start  of  the  vignette  briefing,  subjects  were  given  a  Preparation  Sheet  which 
they  could  use  to  take  notes  on  during  the  briefing  or  as  they  processed  messages.  The  scenario 
briefing  and  overview  of  the  simulator/modified  email  system  took  approximately  30  minutes.  The 
presentation  of  the  training  vignette  required  approximately  10  minutes.  Subjects  were  provided  with 
a  copy  of  the  briefing  materials  as  well  as  the  full  text  of  the  background  situation,  the  oplan,  an 
associated  intelligence  annex,  and  the  vignette  description. 

The  practice  and  feedback  portion  of  training  ensued.  For  the  practice  and  feedback  portions 
of  the  training,  and  for  the  data  collection  scenario,  each  subject  was  paired  with  an  experimenter- 
controller.  During  the  practice  and  feedback  portion  of  training,  the  controller  helped  the  subject  learn 
how  to  use  the  specialized  email  program.  During  both  the  training  and  the  data  collection  scenario, 
the  controller,  playing  the  role  of  other  members  of  the  organization,  responded  to  messages  sent  by 
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the  subject.  The  controller  also  answered  any  questions  the  subjects  had  about  the  scenario,  the  task 
force  organization  used  in  the  scenario,  and  their  role.  The  practice  scenario  was  run  for  15  minutes, 
during  which  38  messages  were  sent  to  the  subjects.  Subjects  were  asked  to  practice  replying  to, 
forwarding  or  generating  new  messages  to  ensure  they  knew  how  to  do  send  out  messages.  Subjects 
in  the  training  group  were  given  feedback  on  their  message  processing  actions. 

The  Post-training  Questionnaire  was  administered  after  the  practice  and  feedback 
components.  Subjects  were  given  as  much  time  as  they  needed  to  complete  it  (approximately 
five  minutes).  One  of  the  experimenters  then  gave  a  briefing  on  the  data  collection  vignette, 
during  which  he  pointed  out  important  locations  on  a  map  overlay.  As  for  the  training  vignette, 
the  briefing  included  the  problem  situation,  the  subject’s  role  in  the  organization,  and  his  situation 
assessment  task.  The  subjects  were  given  a  copy  of  the  vignette  description. 

The  data  collection  scenario  ran  for  30  minutes.  Figure  3-7  depicts  the  data  collection 
process.  At  the  midpoint,  the  scenario  generator  was  paused,  mid  the  subjects  were  asked  to 
complete  the  Briefing  Preparation  form  and  then  to  do  an  oral  sitrep  in  which  they  described  the 
situation  as  they  understood  it  at  that  point.  Following  the  sitrep,  the  scenario  was  resumed  for 
another  15  minutes.  When  it  terminated  subjects  were  told  they  could  use  a  Briefing  Preparation 
form  to  organize  their  thoughts.  They  then  gave  their  final  oral  sitrep. 
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Figure  3-7.  Data  Collection  Process  for  Evaluating  Information  and  Management  Training  Program 
After  completing  their  final  sitrep,  subjects  filled  out  the  Post-Experiment  Questionnaire 
and  then  were  debriefed  by  one  of  the  experimenters.  Subjects  were  invited  to  provide  any 
additional  feedback  they  wanted  on  the  training  or  the  scenario  materials.  Finally,  they  were 
thanked  for  their  participation  and  asked  not  to  discuss  the  experiment  until  all  the  experiment 
sessions  were  completed. 

3.2.5  Participants 

The  training  evaluation  was  conducted  in  the  Futures  Laboratory  of  the  Battle  Command 
Battle  Lab  (BCBL)  at  Fort  Leavenworth,  KS.  The  20  subjects  who  participated  in  the  experiment 
were  randomly  assigned  to  experiment  condition,  with  1 1  receiving  the  IM  training  and  nine 
receiving  the  control  materials.  The  subjects,  all  of  whom  were  affiliated  with  the  BCBL  at  the 
time  they  participated  in  the  evaluation,  included  16  active  duty  officers  and  four  retired  military 
officers.  All  subjects  had  been  in  the  military  for  more  than  1 1  years,  with  the  average  for  both  the 
control  and  trained  subjects  being  18  years.  Two  subjects  held  the  rank  of  captain,  10  were  majors, 
seven  were  lieutenant  colonels,  and  one  was  a  full  colonel.  There  was  no  significant  difference 
between  the  control  and  trained  subjects  in  terms  of  their  rank  or  years  of  service. 
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3-3  TRAINING  EVALUATION:  RESULTS 

In  this  section  we  focus  on  the  results  from  the  data  collection  scenario  that  we  used  to 

evaluate  the  effectiveness  of  the  IM  training  program  that  we  administered.  We  first  present  some 
information  about  the  backgrounds  of  the  officers  who  participated  as  subjects  in  the  experiment. 
Then  we  present  the  analysis  of  the  subjects’  responses  to  the  Post  Training  Questionnaire.  We 
next  examine  the  impact  of  the  training  program  on  the  subjects’  understanding  of  the  situation,  as 
conveyed  in  their  interim  and  final  sitreps.  We  then  turn  to  the  analysis  of  the  message  processing 
to  examine  the  impact  of  the  training  procedure  on  the  efficiency  and  effectiveness  of  the  subjects’ 
information  processing  actions. 

3.3.1  Comprehension  of  Training  Information 

Following  the  practice  and  feedback  portion  of  the  training,  we  administered  a  Post- 

Training  Questionnaire  (see  Appendix  B)  to  assess  whether  the  trained  subjects  comprehended  the 
IM  concepts  and  techniques  that  were  discussed  in  the  lecture  and  demonstration  portions  of  the 
training  program.  To  make  the  questionnaire  seem  appropriate  for  the  control  condition,  we 
included,  but  did  not  score,  questions  about  the  Army  of  the  future  and  the  organizational  structure 
described  in  the  scenario  that  we  used.  In  scoring  the  responses  we  gave  credit  for  answers  that 
reflected  the  procedures  and  techniques  that  were  conveyed  during  training,  including  establishing 
key  words  that  reflect  critical  information  requirements,  filtering  according  to  information  in  the 
message  header,  and  scanning  messages  for  keywords. 

We  allocated  a  total  of  12  points  for  the  three  questions  that  we  scored  (Questions  2,  3,  and 
4).  Trained  subjects  achieved  significantly  higher  scores  (mean  =  8.5)  than  control  subjects  (mean 
=  5.2)  (t=  3.64,  df  =  18,  p=. 001 ').  Using  the  control  mean  as  a  baseline  score,  the  results  indicated 

1  Unless  otherwise  stated,  the  probability  values  a  based  on  a  one-tailed  test  of  significance.  We  used  a  one-tailed  test 
because  we  specified  a  priori  the  direction  of  the  difference  between  means. 
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that  after  the  training  program  was  administered,  trained  subjects  knew  more  about  the  IM 
concepts  and  techniques  that  were  discussed  than  prior  to  training.  The  extent  to  which  trained 
subjects  could  apply  the  concepts  and  techniques  in  a  dynamic  IM  situation  was  assessed  through 
their  performance  and  information  processing  behavior  in  the  evaluation  scenario.  We  turn  to 
those  results  next. 

3.3.2  Performance 

We  hypothesized  that  the  training  procedure  should  enhance  the  ability  of  individuals  to 
recognize  and  comprehend  the  critical  information  conveyed  in  the  message  traffic,  and  understand 
its  implications  for  the  problem  that  was  posed.  The  training  should  also  help  subjects  assess 
what  information  they  are  missing,  and  project  where  and  how  they  can  obtain  that  information. 

To  evaluate  this  aspect  of  the  training,  we  examined  the  subjects  interim  and  final  sitreps. 

An  SME  blind  to  the  experimental  condition  to  which  subjects  were  assigned  scored  the 
subjects’  interim  and  final  sitreps  using  the  Situation  Assessment  Evaluation  Scale  that  we 
developed  (see  Appendix  B).  The  same  form  was  used  for  both  the  written  and  oral  briefings.  The 
first  part  of  the  form  listed  the  important  themes  conveyed  in  the  critical  messages.  For  the  interim 
briefing  we  identified  10  important  themes  reflecting  activities  and  events  reported  in  the  first  half 
of  the  scenario.  For  the  final  briefing,  we  identified  five  themes  reflecting  actions  that  occurred  in 
the  second  half  of  the  scenario.  The  primary  reason  for  having  the  SME  score  the 
presence/absence  of  these  themes  was  to  focus  his  attention  on  factors  he  should  look  for  in  the 
narratives  and  insure  that  he  was  extracting  as  much  as  possible  out  of  the  briefings. 

The  second  part  of  the  form  includes  three  7-point  behaviorally-anchored  scales  reflecting 
three  factors  associated  with  information  processing:  the  overall  quality  of  the  subjects’  situation 
assessment;  the  degree  to  which  subjects  integrated  information  contained  in  the  messages,  and  the 
extent  to  which  subjects  specified  information  seeking  activities  for  improving  SA  and  reducing 
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uncertainty.  The  same  scales  were  used  for  the  interim  and  final  briefings.  The  SME  rated  the 
subjects’  briefings  on  each  of  these  scales,  and  these  ratings  served  as  the  primary  measures  of 
performance. 

Assessment  of  Interim  Briefings 

At  the  time  of  the  interim  briefing,  the  situation  in  the  vignette  was  still  ambiguous.  The 
intent  of  the  Country  A’s  9th  Division  was  still  unclear.  The  evaluations  of  the  briefings  reflected 
the  extent  to  which  subjects  had  picked  up  and  assimilated  the  critical  information  supporting  each 
of  three  alternative  hypotheses  advanced  by  intel  officer  about  the  intent  of  the  Country  A’s 
forces. 

Figures  3-8,  3-9,  and  3-10  show  that  the  trained  subjects  were  consistently  evaluated 
higher  on  the  three  overall  assessment  scales,  both  in  their  written  and  in  their  oral  briefings. 

The  differences  on  their  overall  quality  of  their  situation  assessment  was  significant  for  the 
written  briefing  (t=1.81,  df  =  17,  p  =  .044)  and  marginally  significant  for  the  oral  briefing  (t=1.33, 
df  =  18,  p  =  .099).  The  difference  between  the  two  groups  on  integration  of  information  in  the 
messages  was  significant  in  the  oral  briefings  (t=1.72,  df  =  18,  p  =  .051),  but  not  in  the  written 
briefs.  The  two  groups  did  not  differ  significantly  on  the  extent  and  appropriate  with  which  they 
provided  means  to  improve  their  situation  assessment  and  reduce  uncertainty.  In  part  this  may  be 
because  in  the  instructions  to  the  subjects  we  did  not  explicitly  ask  for  this  information,  and  not  all 
subjects  offered  it  spontaneously.  As  Fig.  3.10  shows,  the  mean  ratings  on  this  scale  were  lower 
than  on  the  other  two  scales  for  both  groups. 


Training  Information  Management 
In  Distributed  Organizations 


63 


ALPHATECH,  INC. 


Figure  3-8.  Quality  of  Situation  Assessment  in  Written  and  Oral  Briefs  at  Interim  Briefing 


Figure  3-9  Quality  of  Information  Integration  in  Written  and  Oral  Briefs  at  Interim  Briefing 
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Figure  3-10  Extent  and  Appropriateness  of  Means  to  Improve  Situation  Assessment  in  Written 

and  Oral  Briefs  at  Interim  Briefing 

Subjects  receiving  training  were  better  able  than  controls  to  integrate  the  critical 
information  in  the  messages  at  the  time  of  the  interim  briefing.  The  higher  quality  integration  of 
information  was  reflected  in  the  quality  of  their  situation  assessments,  which  were  superior  to 
those  of  the  controls. 

Assessment  of  Final  Briefings 

By  the  end  of  the  vignette,  it  was  evident  from  the  messages  that  Countiy  A’s  9  Division 
was  moving  south  toward  the  planned  insertion  area.  Subjects’  briefings  were  evaluated  on  the 
extent  to  which  they  provided  supporting  evidence  for  their  conclusions,  realized  that  the  CTF 
plan  had  been  compromised,  recognized  the  need  for  replanning,  and  specified  additional 
information  requirements  and  potential  sources  for  that  information. 

At  the  time  of  the  final  briefing  virtually  all  the  subjects  had  correctly  assessed  the 
situation.  There  was  no  difference  in  mean  ratings  for  the  trained  and  control  subjects  on  the  three 
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overall  scales.  We  consider  the  possible  reasons  why  the  differences  observed  in  the  interim 
briefings  were  not  evident  at  the  final  briefing  in  Section  3.4. 

3.3.3  Message  Processing:  High-level  Message  Filtering 

High-level  message  filtering  pertains  to  the  initial  filtering  individuals  do  in  deciding 

which  messages  to  attend  to,  and  which  messages  to  ignore,  or  at  least  put  aside  when  processing 
time  is  limited.  The  first  two  components  of  the  procedure  offered  in  the  training  program 
(prepare  and  filter)  combine  to  assist  individuals  in  high-level  filtering.  In  the  preparation  portion, 
individuals  identify  critical  information  they  will  need,  and  from  where  it  is  likely  to  come.  For 
high-level  filtering,  individuals  apply  this  to  incoming  messages.  To  assess  subjects’  high  level 
filtering,  we  looked  at  the  which  messages  subjects  opened,  the  order  in  which  they  opened 
messages,  and  the  their  latency  in  opening  messages  -  i.e.,  how  long  the  messages  sat  in  the  queue 
before  subjects  opened  them. 

We  looked  first  at  the  number  of  messages  opened.  Given  that  more  than  two  messages 
per  minute  were  delivered,  we  anticipated  that  subjects  might  not  have  enough  time  to  open  all 
the  messages.  The  data  confirmed  that  assumption.  On  average,  subjects  opened  only  60  percent 
of  the  80  messages  that  were  delivered,  and  none  of  them  opened  all  the  messages.  We  did  not 
hypothesize  that  the  training  would  have  an  impact  on  the  number  of  messages  opened,  and 
found  none.  The  means  for  the  two  groups  were  virtually  identical  (48.6  for  controls  and  48.1 
for  trained). 

Our  hypotheses  about  differences  between  the  control  and  trained  groups  concerned  which 

messages  subjects  opened.  One  possible  strategy,  albeit  not  an  effective  one,  is  to  go  down  the  list 

of  messages  opening  them  in  the  sequence  in  which  they  appear  in  the  message  listing.  Another 

strategy,  the  one  advocated  in  the  training  program,  is  to  choose  which  messages  to 

open  based  on  the  source,  subject,  and  timing  information  conveyed  in  the  message  header. 
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We  hypothesized  that  trained  subjects  would  be  able  to  assess  the  information  conveyed  in  the 
message  header  in  light  of  the  critical  information  requirements  and  information  sources  they  had 
identified,  and  open  more  critical  messages  and  fewer  nonrelevant  messages  than  control  subjects. 

Table  3-1  shows  the  percentage  of  critical,  relevant,  and  nonrelevant  messages  opened  by 
the  trained  and  control  subjects.  Trained  subjects  opened  a  significantly  higher  percentage  of 
critical  messages  (t  =  2.63,  df  =  18,  p=.009),  a  slightly  higher  percentage  of  the  relevant  messages 
and  a  smaller  percentage  of  noncritical  messages  (t=1.57,  df  =  18,  p  =  .067)  than  the  control 
subjects.  In  other  words,  though  both  groups  opened  the  same  number  of  messages,  trained 
subjects  opened  more  of  the  critical  and  relevant  messages  than  did  the  control  subjects. 


TABLE  3-1.  PROPORTION  OF  CRITICAL,  RELEVANT,  AND  NONRELEVANT 


MESSAGES  OPENED  BY  TRAINED  AND  CONTROl 

SUBJECTS 

Group 

Relevant 

(32  messages)) 

Nonrelevant 

(38  messages) 

.93 

.79 

.36 

.69 

.73 

.48 

Figure  3-1 1  portrays  this  data  in  the  form  of  a  Receiver-Operator  Characteristic  (ROC) 
Curve  (Green  and  Swets,  1974).  The  y-axis  represents  the  proportion  of  critical  messages  opened- 
‘hits’  in  ROC  terminology.  The  x-axis  represents  the  number  of  noncritical  messages  opened  - 
‘false  alarms’  in  ROC  terminology.  The  blue  diagonal  line  represents  cases  in  which 
the  operator  opens  an  equal  proportion  of  critical  and  nonrelevant  messages  -  i.e.,  chance 
performance.  Performance  of  individuals  who  opened  all  the  messages  would  fall  in  the  upper 
right  hand  comer  of  the  graph.  The  hit  rate  would  be  one,  since  they  would  have  opened  100 
percent  of  the  critical  messages,  but  the  false  alarm  rate  would  be  one  as  well,  since  they  would 
also  have  opened  all  the  noncritical  messages.  An  individual  who  had  perfect  performance 
(opening  100  percent  of  critical  messages  and  none  of  the  nonrelevant  messages)  would  be  in  the 
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upper  left  hand  comer.  Figure  3-1 1  shows  that  the  trained  subjects  opened  almost  all  the  critical 
messages  whereas  the  performance  of  the  control  subjects  was  close  to  chance. 
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Figure  3-11.  Proportion  of  Critical  and  Nonrelevant  Messages  (Hits  and  False  Alarms, 
Respectively)  Opened  by  Trained  and  Control  Subjects. 

Data  from  the  post  experiment  questionnaire  supports  the  conclusion  that  trained  subjects 
were  more  selective  about  which  messages  they  opened.  We  asked  subjects  to  assess  what 
proportion  of  the  messages  they  left  open  for  lack  of  time.  The  mean  number  reported  by  trained 
subjects  was  15.8  percent  while  the  mean  for  controls  was  28.2  percent.  Fifty-five  percent  of  the 
trained  subjects,  but  only  33  percent  of  the  control  subjects  said  they  left  less  than  10  percent  of 
the  messages  unopened  because  they  did  not  have  enough  time. 

We  can  also  ask  whether  trained  subjects  were  able  to  make  decisions  more  quickly  about 
which  messages  to  open.  To  assess  this,  we  examined  subjects’  latency  in  opening  messages  -  i.e., 
the  number  of  seconds  that  elapsed  between  the  time  messages  were  received  and  the  time  they  were 
opened.  If  trained  subjects  are  better  prepared  to  filter  out  nonrelevant  messages,  we  would  expect 
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that  they  would  decide  more  rapidly  than  do  control  subjects  which  messages  to  open.  Therefore, 
the  latency  between  time  received  and  time  opened  would  be  smaller  for  trained  subjects. 


Critical  Relevant  Nonrelevant 

Relevance  of  Message 


Figure  3-12  Latency  between  Receipt  of  Message  and  Opening  of  Message  for  Critical,  Relevant, 

and  Nonrelevant  Messages 

Figure  3-12  shows  the  mean  latency  in  message  handling  for  control  and  trained  subjects, 
broken  down  by  actual  relevance  of  the  messages.  The  figure  clearly  shows  that  the  trained 
subjects  opened  all  three  categories  of  messages  more  rapidly  than  did  control  subjects.  Across 
the  three  categories,  the  difference  was  marginally  significant  (t=l  .30,  df  =  18,  p  =  .106).  Given 
that  there  was  no  difference  in  the  total  number  of  messages  opened  by  each  group,  we  cannot 
attribute  the  difference  in  message  processing  latency  to  the  number  of  messages  opened.  The 
pattern  portrayed  in  Figure  3-9  shows  that  the  trained  subjects  opened  critical  and  relevant 
messages  more  rapidly  than  nonrelevant  ones,  whereas  the  control  subjects  tended  to  open 
nonrelevant  messages  more  quickly. 
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3.3.4  Message  Processing:  Content  Filtering 

A  second  level  of  assessment  occurs  when  individuals  open  a  message.  At  this  point,  they 
have  decided  that  the  message  may  contain  pertinent  information  based  on  the  information  in  the 
message  header.  In  the  training  we  emphasized  the  importance  of  identifying  key  words  and 
phrases  as  a  way  of  evaluating  the  importance  of  the  message.  When  the  message  is  opened, 
individuals  must  assess  whether  or  not  it  contains  information  relevant  for  their  mission.  If  they 
determine  the  information  is  not  relevant,  they  should  close  the  message  as  quickly  as  possible. 

If  the  message  is  deemed  important,  they  should  attend  more  carefully  to  the  information  that  it 
contains.  In  the  training  program,  this  second-level  evaluation  was  captured  in  the  ‘scan’  step,  the 
idea  being  that  before  carefully  reading  a  message,  individuals  should  scan  it  to  see  if  it  contains 
key  words  and  phrases  identified  during  the  preparation  phase.  If  subjects  are  effectively  scanning 
messages  to  determine  whether  they  contain  important  mission-related  information,  they  should  be 
likely  close  the  nonrelevant  messages  more  quickly  than  the  critical  ones. 

Message  Processing  Time 

We  hypothesized  that  when  messages  were  opened,  trained  subjects  would  read  and  close 
the  nonrelevant  messages  more  rapidly  because  they  would  have  in  mind  a  set  of  key  words 
signaling  that  the  message  could  be  relevant  for  their  mission,  they  would  scan  for  those  key 
words,  and  would  rapidly  close  the  message  if  those  key  words  were  not  found.  To  test  this 
hypothesis  we  calculated  the  trained  and  control  subjects’  message  processing  time  -  that  is, 
the  amount  of  time  a  message  was  open  -  for  nonrelevant  messages,  but  found  no  differences 
between  the  two  groups.  The  mean  length  of  time  nonrelevant  messages  were  opened  was  23.7 
seconds  for  trained  subjects,  slightly  higher  than  the  mean  for  control  subjects,  20.7  seconds. 

Nor  were  there  any  differences  between  the  two  groups  on  the  processing  time  for  critical  and 
relevant  messages. 
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Not  all  individuals  are  able  to  scan  text  effectively,  and  it  is  possible  that  this  is  a  skill  that 
needs  to  be  honed  before  it  can  be  effectively  applied  as  a  deliberate  message  processing  strategy. 
This  conjecture  gains  some  support  from  data  in  the  post  experiment  questionnaire.  Over  50 
percent  of  the  trained  subjects  but  less  than  20  percent  of  the  control  subjects  mentioned  using 
key  words  as  a  criterion  for  scanning.  In  other  words,  significantly  more  of  the  trained  subjects 
than  controls  asserted  that  they  were  using  key  words  to  scan  the  text.  But  the  message  processing 
time  data  does  not  provide  evidence  that  the  trained  subjects  actually  were  scanning  the  text.  It  is 
also  possible  that  the  amount  of  time  a  message  was  left  open  may  not  be  a  sensitive  indicator  of 
whether  or  not  individuals  were  scanning  the  text.  Another  kind  of  measure  may  be  required. 

Message  Classification 

We  hypothesized  that  the  trained  subjects  would  be  better  able  to  evaluate  the  importance 
of  the  messages  they  read.  To  test  this  hypothesis  we  examined  the  subjects’  ratings  of  the 
messages  they  opened.  Table  3-2  presents  the  proportion  of  messages  subjects  opened  that  they 
correctly  categorized  as  critical,  important,  and  nonrelevant.  The  difference  between  the  trained 
and  control  subjects’  ability  to  correctly  classify  critical  messages  they  opened  was  marginally 
significant  (t=l  .36,  df  18,  p=095),  but  the  differences  for  relevant  and  nonrelevant  messages 
were  nonsignificant. 

An  interesting  feature  evident  in  Table  3-2  is  that  within  each  relevance  category  subjects 
correctly  classified  only  slightly  more  than  half  the  messages.  Close  to  half  were  misclassified. 

We  examined  the  nature  of  the  misclassifications  to  see  whether  they  were  confusions  between 
adjacent  categories  (e.g.,  critical/relevant  and  relevant/noncritical),  or  were  errors  of  extremes 
(e.g.,  critical  called  nonrelevant  and  nonrelevant  called  critical)? 
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TABLE  3-2.  PROPORTION  OF  CORRECTLY  CLASSIFIED  CRITICAL,  RELEVANT,  AND 
NONRELEVANT  MESSAGES  OPENED  BY  TRAINED  AND  CONTROL  SUBJECTS 


Group 

Critical 

Relevant 

Nonrelevant 

.65 

.53 

.59 

.52 

.55 

.52 

Figure  3-13  addresses  this  question.  It  portrays  the  trained  and  control  subjects’  ratings  of 
the  critical  and  noncritical  messages  they  opened  in  the  form  of  an  ROC  curve.  The  hits  (y-axis) 
represent  critical  messages  rated  as  critical  and  the  false  alarms  (x-axis)  represent  one  kind  of 
extreme  misclassification,  noncritical  messages  rated  as  critical.  Overall,  the  trained  subjects  were 
better  able  to  identify  critical  messages  and  had  fewer  false  alarms  than  the  control  subjects.  The 
false  alarm  rate  was  quite  low  in  both  groups  (.04  for  trained  and  .11  for  control  subjects), 
suggesting  that  when  a  nonrelevant  message  was  opened,  it  was  unlikely  that  an  individual 
mistook  it  as  being  critical.  But  this  low  false-alarm  rate  was  reflected  in  hit  rates  that  were  only 
moderately  high.  The  trained  subjects  identified  about  two-thirds  and  the  control  subjects  one-half 
of  the  critical  messages  they  opened,  indicating  that  some  critical  information  was  not  recognized 
by  subjects  in  both  groups. 

To  further  understand  the  nature  of  the  classification  mistakes,  we  also  calculated  how 
often  critical  messages  were  called  nonrelevant,  and  found  an  even  lower  error  rate.  On  average, 
under  three  percent  of  the  critical  messages  opened  were  called  nonrelevant,  and  there  was  no 
difference  between  trained  and  control  subjects  in  this  regard.  For  messages  they  opened,  then, 
subjects  were  unlikely  to  totally  miss  critical  information.  If  they  didn’t  rate  it  as  critical,  they  at 
least  identified  it  as  relevant.  The  difficulty  seems  to  be  in  distinguishing  critical  from  relevant 
messages. 
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Figure  3-13.  ROC  Curve  for  Messages  Opened 
Finally,  we  looked  at  how  subjects  rated  relevant  messages.  Table  3-2  shows  that  just  over 
half  the  relevant  messages  were  correctly  classified.  What  about  the  remainder?  Were  they  more 
likely  to  be  mistaken  as  critical  or  as  nonrelevant?  We  found  that  most  relevant  messages  were 
called  either  relevant  or  critical.  Few  of  the  relevant  messages  (13  percent  for  trained  and  1 5 
percent  for  controls)  were  mistaken  as  nonrelevant.  In  other  words,  both  the  trained  and  control 
subjects  were  unlikely  to  dismiss  relevant  messages;  their  problem  was  in  distinguishing  the 
critical  information,  and  the  training  did  not  have  an  effect  in  this  regard. 


3.3.5  Message  Generation 

One  of  the  key  concepts  embodied  in  the  training  program  was  that  individuals  must 
be  informed  about  the  organization  in  which  they  are  embedded.  The  training  emphasized  the 
need  to  consider  where  critical  information  would  be  likely  to  come  from,  and  also  the  kinds  of 
information  that  other  nodes  would  need.  The  training  program  also  stressed  that  a  major  source 
of  information  overload  is  unnecessary  message  traffic  generated  by  individuals  who  forward 
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messages  globally,  or  forward  messages  without  modifying  them  so  as  to  make  the  critical 
information  salient  for  the  intended  recipient.  This  component  of  the  training  program  should 
sensitize  the  trained  subjects  and  help  make  them  more  effective  information  seekers  and 
distributors.  We  assessed  the  effectiveness  of  this  aspect  of  the  training  program  by  looking 
at  the  messages  generated  by  the  subjects. 

Subjects  were  able  to  reply  to  messages,  to  forward  messages  with  or  without 
modifications,  and  to  generate  new  messages.  We  looked  at  the  number,  types,  and  destinations 
of  messages  sent  by  subjects  to  other  nodes  in  the  organization. 

Message  Volume 

We  looked  first  at  the  overall  number  of  messages  that  were  sent.  The  training  emphasized 
that  individuals  may  unwittingly  contribute  to  information  overload  by  sending  out  unnecessaiy  or 
irrelevant  information,  or  by  broadcasting  information  too  widely.  If  the  training  was  in  making 
individuals  careful  about  the  messages  they  send  out,  we  would  expect  that  the  volume  of 
messages  sent  by  trained  subjects  would  be  lower  than  the  volume  sent  by  control  subjects. 

The  data  showed  that  trained  subjects  sent  out  an  average  of  8.4  messages  while  control 
subjects  sent  out  an  average  of  13.7  messages,  more  than  half  again  as  many.  The  difference 
between  the  two  groups  was  marginally  significant  (t=  1.30,  df  =  18,  p  =  .106).  The  results 
suggest  that  trained  subjects  were  more  careful  about  messages  they  generated.  This  attribution 
gains  support  from  data  in  the  post  experiment  questionnaire.  More  trained  than  control  subjects 
indicated  they  did  not  send  information  to  other  nodes,  and  the  trained  subjects  were  more  likely 
than  controls  to  say  that  it  was  because  it  wasn’t  necessary  to  send  any  information  to  other  nodes. 
Similarly  trained  subjects  reported  requesting  information  less  frequently  from  other  nodes  than 
did  control  subjects.  When  asked  why  they  didn’t  request  information,  trained  subjects  were  more 
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likely  to  say  it  was  because  they  didn’t  need  any  information,  whereas  controls  were  more  likely  to 
say  it  was  because  they  didn’t  know  where  to  get  it  or  didn’t  know  what  to  ask. 

Messages  generated  by  subjects  include  three  types:  replies  to  messages  received;  messages 
forwarded  to  other  nodes;  and  new  messages.  Figure  3-14  shows  that  few  new  messages  were 
generated  by  either  control  or  trained  subjects.  For  both  groups,  the  majority  of  the  messages 
generated  were  forwards.  Control  subjects  replied  to  a  slightly  larger  proportion  of  the  messages 
whereas  trained  subjects  forwarded  more  messages,  but  the  difference  was  nonsignificant. 


Figure  3-14.  Distribution  of  Types  of  Messages  Generated  by  Trained  and  Control  Subjects 
More  significant  than  the  number  or  type  of  messages  generated  by  subjects  is  the  nature  of 
the  messages  that  were  sent  out  -  do  they  contain  relevant  or  nonrelevant  information.  Virtually  all 
generated  messages  were  replies  or  forwards,  and  both  of  these  types  are  based  on  a  source  message 
that  was  received  by  the  subject.  Figure  3-1 5  shows  the  percentage  of  replies  and  forwarded 
messages  that  stemmed  from  critical,  relevant,  and  nonrelevant  source  messages.  The  proportion 
of  messages  sent  by  trained  subjects  that  pertained  to  critical  source  messages  was  significantly 
higher  than  the  proportion  sent  by  controls  (t=2.24,  df  =  18,  p  =  .019),  and  conversely,  the 
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proportion  of  messages  that  pertained  to  relevant  source  messages  was  significantly  higher  for 
control  subjects  (t  =  1.94,  df  =  18,  p  =  .034).  Both  groups  of  subjects  generated  few  messages  that 
were  based  on  nonrelevant  source  messages.  These  results  indicate  that  trained  subjects  were  more 
attuned  to  critical  information  in  the  message  traffic  they  generated. 


Critical  Relevant  Nonrelevant 

Relevance  of  Source  Information 


Figure  3-15  Percentage  of  Messages  Generated  based  on  Critical,  Relevant,  and  Nonrelevant 

Source  Messages 

Depth  of  Message  Processing 

To  shed  further  light  on  how  training  impacted  the  nature  of  messages  that  were 
transmitted,  we  examined  the  amount  of  processing  that  was  done  on  the  messages  that  were 
sent  out.  To  measure  depth  of  message  processing,  we  created  three  categories  of  messages: 

1)  messages  that  were  simply  forwarded  to  another  node  without  any  modification;  2)  messages 
that  were  modified  in  some  way,  and;  3)  newly  created  messages.  We  hypothesized  that  the 
trained  subjects  would  be  more  likely  than  control  subjects  to  modify  forwarded  messages  or 
generate  new  ones,  but  the  data  did  not  support  this  hypothesis.  As  shown  in  Figure  3-14,  few 
new  messages  were  generated  by  either  group.  Of  the  forwarded  messages  trained  and  control 

Training  Information  Management  76 

In  Distributed  Organizations 


ALPHATECH,  INC. 


subjects  were  equally  likely  to  forward  the  message  without  any  modifications  or  additions.  This 
result  may  in  part  reflect  the  trained  subjects’  belief  that  the  entire  message  was  relevant,  and  there 
was  no  need  to  summarize  or  restate  it,  but  it  also  suggests  that  the  aspects  of  the  training  program 
that  help  individuals  minimize  unnecessary  information  in  messages  could  be  bolstered. 

Message  Destination 

When  subjects  forwarded  messages,  they  could  send  them  to  single  node  or  to  multiple  node 
destinations.  For  example,  a  message  sent  to  the  aviation  brigade  goes  to  that  single  node.  A  message 
sent  to  Force  IV  goes  to  all  the  nodes  in  Force  IV.  A  message  to  multiple  nodes  is  more  likely  to  be  a 
source  of  information  overload  because  not  all  the  nodes  receiving  the  message  may  require  the 
information  it  contains.  The  data  showed  that  control  subjects  were  more  likely  to  send  one  or  more 
of  their  messages  to  multiple  nodes  than  were  the  trained  subjects,  with  56  percent  the  control  and  44 
percent  of  the  trained  doing  so.  The  difference  is  not  significant,  but  does  point  to  slightly  more 
caution  and  care  on  part  of  trained  subjects  in  terms  of  how  widely  they  broadcast  their  messages. 

We  examined  the  level  in  the  organization  to  which  messages  were  sent.  Trained  subjects 
sent  more  messages  to  higher  authority  than  laterally  or  to  subordinates.  That  finding  is  consistent 
with  the  expectation  of  the  SME  who  designed  the  scenario  who  similarly  suggested  more 
messages  that  were  appropriate  for  direction  to  higher  authority  than  messages  that  should  be 
directed  to  lateral  or  subordinate  units. 

3.4  TRAINING  EVALUATION:  CONCLUSIONS  AND  DISCUSSION 
3.4.1  Conclusions 

The  training  program  was  designed  to  help  individuals  be  efficient  and  effective  information 
managers.  The  evaluation  experiment  was  designed  to  assess  the  effectiveness  of  the  training 
program  by  analyzing  participants’  processing  of  information  and  their  situation  assessments. 

We  examined  participants’  attention  to  incoming  messages,  their  evaluations  the  criticality  of  the 
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messages  and  their  processing  of  outgoing  messages.  We  assessed  the  degree  to  which  they 
comprehended  and  integrated  the  messages  they  read  from  their  written  and  oral  sitreps. 

The  performance  results  showed  that  the  trained  subjects’  ability  to  integrate  and  assess 
critical  mission  information  was  superior  to  that  of  the  controls.  The  effectiveness  of  the  training 
was  evidenced  in  the  interim  evaluation,  the  time  at  which  the  situation  was  most  fluid  and 
uncertain,  and  was  reflected  in  the  higher  quality  of  the  trained  subjects’  situation  assessment  and 
their  integration  of  information  in  the  messages. 

The  information  processing  data  provided  behavioral  evidence  for  the  effectiveness  of  the 
training  program,  and  indicated  why  the  performance  of  the  trained  subjects  was  superior  to  that  of 
the  controls.  On  average,  trained  and  control  subjects  opened  the  same  number  of  messages,  but 
the  trained  subjects  opened  messages  more  quickly  than  control  subjects.  In  the  experiment,  we 
observed  that  whereas  a  control  subject  might  move  the  cursor  up  and  down  the  list  of  messages 
while  deciding  which  message  to  open  next,  trained  subjects  were  more  definitive  in  the  selection 
process,  rapidly  bypassing  those  messages  that  were  nonrelevant  and  opening  those  that  might  be 
relevant.  These  data  and  observations  suggest  that  the  preparation  component  of  the  training  was 
successful  in  helping  subjects  clarify  what  critical  information  they  needed,  and  from  were  the 
information  might  come,  and  therefore  they  spent  less  time  deciding  which  messages  to  open. 
Consistent  with  this  inference,  we  found  that  the  trained  subjects  opened  a  higher  proportion  of 
the  critical  messages  and  a  smaller  proportion  of  the  nonrelevant  messages  than  the  controls. 
Trained  subjects  were  also  more  likely  to  correctly  classify  critical  messages  they  opened  than 
control  subjects. 

Trained  subjects  were  more  careful  information  generators.  They  generated  fewer 
messages  than  controls,  were  more  likely  to  distribute  them  to  single  rather  than  multiple  nodes, 
and  more  often  directed  them  to  the  appropriate  level  in  the  hierarchy.  In  other  words,  they  saw 
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more  of  the  critical  information  and  had  more  time  to  read  and  integrate  the  messages,  in  part 
because  they  opened  messages  more  rapidly  than  controls  and  in  part  because  they  spent  less  time 
generating  message  traffic. 

3.4.2  Discussion 

Significant  components  of  both  the  performance  and  the  information  processing  results 
showed  that  the  training  program  was  effective  in  improving  the  trained  subjects’  IM  skills.  There 
were,  however,  certain  aspects  of  the  results  that  did  not  confirm  differences  between  the  trained 
and  control  groups. 

As  noted  previously,  virtually  all  the  subjects  solved  the  problem  that  was  posed  in  the 
vignette  -  i.e.,  the  intentions  of  the  Country  A’s  9th  Division.  Given  the  significant  differences 
between  the  trained  and  control  groups’  performance  in  the  interim  briefings,  we  attribute  the 
inability  to  detect  a  difference  in  the  final  sitreps  to  a  ceiling  effect  caused  by  a  too  simple 
resolution  of  the  problem  in  the  second  part  of  the  evaluation  vignette.  A  more  sensitive 
evaluation  would  have  been  obtained  had  more  uncertainty  remained  in  the  situation.  With  little 
or  no  ambiguity  present  by  the  end  of  the  vignette  the  performance  of  all  the  subjects  became 
quite  similar  -  any  differences  were  too  subtle  to  detect  in  the  assessment  scales  we  used. 

One  of  the  behavioral  components  in  the  training  program  advocated  scanning  messages 
that  are  opened  for  key  words  and  phrases  identified  in  the  preparation  phase.  Both  the  post¬ 
training  and  post-experiment  questionnaire  data  showed  that  the  subjects  understood  the  concepts 
of  identifying  key  words  and  scanning  for  those  key  words  in  the  messages.  The  latency  data, 
however,  did  not  provide  evidence  that  trained  subjects  were  more  likely  than  controls  to  use  the 
scanning  technique.  The  lack  of  evidence  for  scanning  may  have  two  explanations.  It  may  be 
that  the  measure  we  used,  the  length  of  time  messages  were  left  open,  was  not  appropriate  for 
differentiating  between  scanning  and  reading.  Alternatively,  it  may  be  the  case  that  although 
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trained  subjects  were  aware  that  they  should  scan  the  messages,  scanning  is  a  skill  that  many  of 
them  do  not  have. 

Some  facets  of  the  message  generation  data  showed  that  the  training  was  effective  in 
limiting  the  transmission  of  nonrelevant  information.  For  example,  messages  forwarded  by  trained 
subjects  were  more  likely  to  pertain  to  a  critical  source  message  than  were  the  messages  generated 
by  control  subjects.  However,  trained  subjects  were  as  likely  as  controls  to  forward  messages 
without  modification.  Clearly  it  is  easier  and  quicker  to  forward  a  message  wholesale  than  to 
revise  it.  The  results  indicate  the  need  to  bolster  aspects  of  the  training  to  encourage  tailoring  the 
message  content  for  the  intended  recipient. 

We  feel  the  most  appropriate  target  audience  for  the  IM  training  would  be  military  officers 
who  have  newly  been  exposed  to  the  MDMP  -  before  they  have  had  a  chance  to  establish  their  own 
IM  style  and  process.  It  is  a  much  more  lengthy  and  difficult  process  to  break  down  old  behaviors 
before  building  up  new  behaviors,  and  the  training  program  we  developed  did  not  incorporate 
procedures  to  do  such  a  thing.  The  individuals  who  participated  in  this  study  are  (or  had  been) 
mid-career  officers,  all  of  whom  already  had  intensive  training  on  information  management,  and 
many  of  whom  had  extensive  experience  as  information  managers  in  situations  similar  to  the  one 
portrayed  in  the  training  evaluation  vignette.  Both  of  these  long  term  factors  probably  had  a 
mitigating  effect  on  the  impact  of  a  two-hour  training  program.  That  differences  between  the  two 
groups  did  emerge  attests  to  the  effectiveness  and  promise  of  the  training  program  we  administered. 

The  comments  of  one  of  the  participants  who  said  he  has  worked  in  systems  with  large 
numbers  of  messages  in  the  queue  supports  our  belief  that  the  training  we  provided  is  on  the  mark 
for  military  officers.  He  remarked  that  the  training  method  suggested  is  just  like  what  he  does,  But 
he  has  never  before  categorized  it  in  that  way.  He  felt  the  training  was  helpful  in  that  it  provided  an 
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organizational  framework  for  what  he  has  been  doing.  We  believe  that  providing  this  framework  is 
itself  a  means  of  encouraging  and  reinforcing  effective  information  management. 
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CHAPTER  4 

CONCLUSIONS  AND  IMPLICATIONS  FOR  INFORMATION  MANAGEMENT 

TRAINING 

Four  interrelated  activities  were  conducted  in  this  project:  1)  iterative  development  of  a 
model  of  information  processing;  2)  conduct  of  an  experiment  to  assess  the  impact  of  differing 
levels  of  information  load  and  different  levels  of  organizational  knowledge;  3)  development  and 
evaluation  of  the  preliminary  version  of  an  information  management  (IM)  training  program,  and; 

4)  development  and  evaluation  of  a  test  version  of  the  IM  training  program.  In  the  first  section  of 
this  chapter  we  show  the  ways  in  which  the  IM  training  prevented  information  processing  errors 
that  have  been  observed.  In  the  second  section  we  present  a  modified  version  of  the  IM  model  that 
incorporates  the  findings  from  the  two  training  evaluations  we  conducted.  In  the  third  section  we 
address  suggested  enhancements  and  extensions  to  the  IM  training  program  and  to  the  materials 
used  to  evaluate  its  effectiveness. 

4.1  PROJECT  SUMMARY 

As  a  result  of  the  work  discussed  in  the  previous  chapters,  we  have  gained  valuable 
insights  into  the  original  model  of  information  management.  Figure  4-1  provides  an  annotated 
version  of  the  IM  model  that  we  used  as  a  basis  for  the  IM  training  program  to  show  which  areas 
of  the  model  were  most  influenced  by  the  training  program.  The  model  was  developed  on  the 
basis  of  theories  of  cognitive  processing  and  observations  of  information  processing  in  naturalistic 
and  experiment  settings.  The  annotations  in  the  figure  show  information  processing  errors 
identified  in  observations  made  in  naturalistic  settings  and  in  the  experiment  we  conducted  to 
assess  the  impact  of  information  load  (referred  to  as  Experiment  1  in  the  figure).  The  green  blocks 
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in  the  figure  show  that  the  training  program  we  developed  addressed  all  of  the  areas  where  typical 
errors  occur.  The  numbers  in  the  blocks  correspond  to  the  numbers  in  the  explanatory  text  below. 
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Figure  4-1.  Areas  of  Training  Influence  on  Information  Processing  Errors 
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The  first  area  of  influence  (see  number  1  in  Figure  4.1)  is  at  what  is  considered  the  high 
level  filter  -  that  part  of  managing  information  where  the  manager  categorizes  the  incoming 
information  as  “potentially  important  and  deserving  of  a  more  in-depth  look”  or  “not  relevant” 
based  on  the  high  level  information  attributes  that  are  apparent  when  information  first  arrives  (e.g., 
with  email,  such  information  would  be  the  source,  subject,  and  date).  Typical  errors  that  occur  at 
this  stage  of  managing  information  are  pursuing  the  wrong  information,  disregarding  information, 
and  not  recognizing  critical  information.  As  a  result  of  the  training  program  the  participants 
opened  fewer  nonrelevant  messages,  more  critical  messages,  and  opened  the  critical  and  relevant 
messages  more  quickly  than  control  participants  did. 

The  second  area  of  influence  (see  number  2),  the  second  level  filter  in  the  IM  model,  was 
also  positively  affected  by  the  training  program.  Specifically,  the  trained  participants  more 
accurately  rated  critical  messages  after  they  were  opened  than  the  control  participants  did.  We 
were  not  able  to  ascertain  directly  whether  the  trained  participants  used  an  updated  assessment  of 
the  situation  to  evaluate  and  revise  their  collection  strategy  (i.e.,  to  revise  the  high-level  filter); 
however  the  finding  that  the  trained  participants  had  a  better  interim  assessment  of  the  situation 
suggests  that  they  were  more  prepared  to  do  so  if  they  felt  it  was  necessary. 

Our  study  also  revealed  evidence  that  the  training  program  positively  affected  the  third 
area  of  influence  noted  by  the  number  3  in  Figure  4-1 .  During  the  study  control  participants 
reported  that  they  did  not  seek  information  because  they  didn’t  have  time,  whereas  the  trained 
participants  sought  little  information  because  they  felt  that  they  did  not  need  any  information. 
These  findings  suggest  that  the  control  participants  experienced  the  failure  to  seek  additional 
information,  whereas  the  trained  group  realized  that  they  could  seek  information  if  necessary,  but 
they  simply  did  not  find  it  necessary.  It  is  possible  that  because  the  trained  subjects  opened 
messages  more  quickly  and  opened  a  higher  proportion  of  critical  messages,  they  felt  more 
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confident  that  they  were  receiving  all  of  the  information  they  needed,  and  concluded  that  it  was 
not  necessary  or  prudent  to  query  other  nodes  for  any  additional  information  that  might  be 
available. 

The  fourth  area  of  influence  (number  4)  concerns  refraining  from  contributing  to  other 
people’s  information  overload.  As  a  result  of  the  training,  trained  participants  sent  fewer 
messages,  a  greater  proportion  of  which  were  critical,  to  other  nodes  in  the  organization.  They 
sent  messages  to  fewer  nodes  at  once,  and  were  more  likely  to  send  the  information  to  the 
appropriate  level  in  the  organizational  hierarchy.  These  findings  suggest  that  the  training  resulted 
in  behaviors  that  would  lessen  the  possibility  that  other  information  processors  would  experience 
information  overload. 

The  final  area  of  influence  (see  number  5)  was  evident  at  the  interim  brief.  The  trained 
participants  were  better  able  to  assess  the  situation  (e.g.,  what  they  knew  about  the  situation  and 
what  gaps  still  existed)  halfway  through  the  scenario  than  the  control  group. 

As  these  findings  suggest,  the  training  program  was  quite  effective  in  counteracting  many 
of  the  typical  errors  that  occur  in  information  management.  Some  errors  noted  in  Figure  4-1 ,  such 
as  continuing  to  pursue  information  when  there  is  not  sufficient  time,  were  not  pertinent  in  the 
vignette  that  we  used  to  assess  the  training  effectiveness,  and  therefore  the  effectiveness  of  the 
training  program  in  preventing  them  could  not  be  assessed.  We  discuss  this  further  in  Section  4.3, 
in  the  context  of  potential  refinements  and  additions  to  the  scenario. 

4.2  ENHANCEMENTS  TO  INFORMATION  MANAGEMENT  MODEL 

Figure  4-2  illustrates  the  IM  model  that  has  been  a  guiding  factor  throughout  this  project. 

To  the  left  is  the  original  model  with  areas  of  most  development  highlighted  with  the  red  boxes. 
To  the  right  is  the  model  as  it  has  evolved  over  the  course  of  the  project.  As  this  figure  illustrates, 
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the  major  changes  in  the  model  have  less  to  do  with  additions  or  deletions  of  steps  in  the  IM 
process  and  more  to  do  with  the  expansion  of  the  steps  as  originally  presented.  The  major  changes 
are  described  in  turn  below. 

One  new  addition  to  the  model  is  the  initial  preparation  step  shown  in  the  top  of  the  revised 
IM  model.  In  the  initial  model,  this  step  was  implicit  -  it  was  simply  assumed  that  effective 
information  managers  would  take  the  time  to  prepare  before  becoming  engulfed  in  their  IM  task. 
Throughout  the  project,  however,  it  became  clear  that  preparation  is  a  key  step  in  managing 
information  that  is  overlooked  -  or  actively  shunned  -  by  poor  information  managers.  Because  it 
is  so  essential  to  the  successful  management  of  information,  we  have  added  it  as  the  initial  step  of 
the  IM  process. 

This  initial  preparation  is  not  intended  to  be  a  burden  to  the  information  manager;  on  the 
contrary,  it  is  thought  of  as  a  cursory  “thought  organization”  that  will  result  in  a  more  efficient  IM 
process.  Preparation  entails  actively  thinking  of  the  commander’s  intent,  considering  what 
information  is  needed  (e.g.,  key  words  and  phrases),  where  that  information  might  come  from, 
where  information  received  might  go  (both  of  which  require  knowledge  of  the  organization),  and 
how  to  determine  the  importance  of  information  received  from  the  commander’s  intent.  This 
quick  preparation  primes  the  information  manager  so  that  important  information  is  more  quickly 
recognized  and  nonrelevant  information  can  be  quickly  discarded.  Many  experts  practice  this 
preparation  step  without  thought;  however,  nonexpert  information  managers  do  not  instinctively 
see  the  benefit  to  taking  the  few  minutes  necessary  to  prepare  for  information  management  and 
they  must  be  trained  to  do  so. 

After  the  initial  preparation,  the  information  manager  must  apply  a  high  level  filter  to  the 
information  being  received.  One  of  the  assumptions  of  this  IM  model  is  that  information 
managers  receive  considerably  more  information  than  they  can  comfortable  consume  in  the  time 
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allotted,  so  this  first  information  filter  involves  categorizing  the  information  (e.g.,  “could  be 
important  and  deserves  a  more  in-depth  look”  vs.  “not  relevant”)  based  on  the  high  level 
information  attributes  that  are  apparent  when  information  first  arrives  (e.g.,  with  email,  such 
information  would  be  the  source,  subject,  and  date).  The  actual  filter  is  derived  from  the  key 
information  requirements  determined  in  the  preparation  stage,  and  only  those  messages  that  pass 
this  first  filter  should  be  looked  at  in  more  detail. 

Those  messages  deemed  “potentially  important”  should  then  be  opened  and  the 
information  manager  should  quickly  scan  for  the  relevant  key  words  and  phrases  (as  determined  in 
the  preparation  step).  Only  those  messages  containing  the  important  key  words  and  phrases 
should  be  read  in  more  detail,  and  only  those  messages  that  are  read  are  used  in  further  actions. 

Another  addition  to  the  IM  model  is  the  feedback  loop  connecting  “scan  and  read”  to  the 
initial  filter.  Information  management  is  a  dynamic  process,  an  aspect  that  the  initial  model  did  not 
capture,  so  this  feedback  loop  was  added  to  represent  the  adaptable  quality  of  managing 
information  through  which  incoming  information  may  result  in  a  revision  of  the  initial  filter.  It  is 
only  with  a  process  such  as  this  that  efficient  information  management  can  take  place. 

A  final  addition  to  the  model  is  the  impact  organizational  knowledge  has  on  the 
information  management  process.  Knowledge  of  the  people  within  the  organization,  and  their 
communication  needs  and  styles,  is  necessary  for  establishing  an  effective  initial  filter  (who  might 
send  the  needed  information),  determining  what  to  do  with  the  information  once  it  is  received,  and 
knowing  where  to  seek  for  information  that  is  still  needed.  Without  a  solid  understanding  of  the 
organization,  effective  information  management  is  improbable.  The  more  likely  result  is  that  the 
information  manager  contributes  to  information  overload. 
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Figure  4-2.  The  Updating  of  the  Information  Management  Model 

4.3  FUTURE  REFINEMENTS  AND  DIRECTIONS 
4.3.1  IM  Training  Program 

The  test  version  of  the  IM  training  program  was  successful  in  many  regards.  But  each 
administration  of  an  evolving  training  program  reveals  refinements  and  new  options  to  make  the 
program  even  stronger  and  more  robust.  Observations  of  the  training  program  as  it  was 
conducted,  observations  during  the  evaluation  phase,  and  comments  made  by  the  participants  are 
additional  sources  of  data  that  supplement  the  findings  based  on  the  analysis  of  the  performance 
and  processing  data.  All  of  these  sources  together  suggested  additional  modifications  and 
enhancements  to  further  strengthen  the  IM  training  program.  We  consider  some  of  these  here. 

Several  of  the  more  experienced  participants  noted  that  they  are  already  trained  to  receive 
the  Commander’s  Critical  Information  Requirements  (CCIRs).  To  insure  that  the  training  does  not 
appear  redundant  with  what  has  already  been  taught,  the  trainer  could  stress  more  heavily  in  the 
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lecture  portion  that  although  officers  are  already  trained  to  receive  CCIRs,  the  thrust  of  this 
training  program  is  to  provide  heuristics  that  are  robust  to  factors  that  degrade  cognitive 
functioning,  such  as  fatigue,  time  pressure,  or  adverse  environmental  conditions. 

The  test  version  of  the  training  program  promoted  the  idea  of  scanning  a  message  to  assess 
its  relevance.  In  constructing  the  training  materials,  we  assumed  that  participants  were  efficient 
scanners,  but  we  now  believe  this  assumption  was  probably  mistaken.  The  measure  we  used 
(amount  of  time  messages  were  left  open)  did  not  provide  any  evidence  for  scanning.  We  suggest 
that  in  the  future,  IM  training  should  incorporate  a  classroom-type  segment  in  which  all  trainees 
are  taught  how  to  scan  more  effectively  and  efficiently. 

In  addition  to  any  training  that  is  provided  in  how  to  scan  text,  to  more  effectively  convey 
applications  of  message  scanning,  the  demonstration  portion  of  the  training  program  should 
include  a  greater  number  of  examples  in  which  a  message  is  opened,  scanned  for  key  words  and 
phrases,  and  deemed  not  relevant,  as  well  as  examples  in  which  key  words  are  detected  and  the 
message  is  read  more  carefully. 

As  we  show  in  the  updated  IM  model  in  Figure  4-2,  information  management  is  a  dynamic 
process.  The  training  could  be  enhanced  by  an  increased  emphasis  on  dynamically  changing 
information  needs.  Examples  in  the  demonstration  and  practice  portions  of  training  that  lead  to 
adding  new  key  words,  or,  conversely,  deleting  a  key  word,  would  emphasize  the  dynamic  nature 
of  information  management.  Similarly  examples  in  which  the  most  likely  source  of  critical 
information  changes  over  time  would  reinforce  awareness  of  the  dynamic  nature  of  information 
management  and  the  need  for  organizational  knowledge. 

The  finding  in  the  evaluation  of  the  test  version  of  the  IM  training  program  that  most  of  the 
messages  that  were  transmitted  were  forwards  of  an  existing  message,  rather  than  newly  composed 
messages  or  modified  forwards,  suggests  that  the  effectiveness  of  the  training  program  could  be 
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enhanced  by  material  that  reinforces  efficient  information  push.  We  did  address  this  in  the 
training  program,  but  in  subsequent  training  we  would  emphasize  this  facet  of  information 
management  with  additional  examples  of  succinct  and  focused  messages  in  the  demonstration 
portion. 

We  observed  during  the  training  evaluation  that  participants  did  not  always  pay  attention  to 
who  were  the  other  recipients  of  messages  they  received.  We  noted  several  instances  in  which 
subjects  forwarded  a  message  to  a  node  that  had  already  received  it,  as,  for  example,  forwarding  to 
a  specific  node  in  a  unit  a  message  that  was  originally  addressed  to  all  nodes  in  the  unit  The 
training  should  stress  the  importance  of  noting  who  has  already  received  the  message,  particularly 
in  cases  where  the  message  is  sent  to  a  group  composed  of  multiple  nodes.  Attention  to  the  list  of 
recipients  needs  to  be  emphasized  in  the  lecture  and  exemplified  in  the  demonstration  portion  as 
something  to  pay  attention  to. 

In  the  administration  of  the  training  program,  we  placed  a  lot  of  emphasis  on  identifying 
key  words,  and  we  exemplified  this  in  the  demonstration  phase.  In  part  because  the  amount  of 
time  available  for  training  was  limited,  we  could  not  follow  through  to  a  sufficient  degree  during 
practice  and  feedback.  If  possible,  in  future  administrations  it  would  be  preferable  to  allot  more 
time  to  the  practice  and  feedback  portion  of  the  training  so  that  the  trainer  could  spend  more  time 
talking  with  the  participants  about  whether  they  were  appropriately  filtering  the  messages,  both  in 
terms  of  high  level  and  secondary  filtering.  The  trainer  needs  to  actively  query  the  participants 
about  the  keywords  they  are  using  to  be  sure  participants  have  enumerated  them,  and  to  provide 
feedback  to  the  participants  on  whether  they  are  applying  the  key  words  appropriately. 

The  testing  methodology  used  in  the  program  was  not  designed  to  assess  the  durability  or 
robustness  of  the  training  program  over  time.  For  this  project  we  trained  participants  and  then 
assessed  the  effectiveness  of  the  training  immediately  thereafter.  Although  we  have  strong 
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evidence  that  the  training  positively  affected  IM  activities  that  occurred  immediately  following  the 
training,  we  do  not  know  whether  the  positive  impacts  of  the  training  will  endure  over  time. 

Ideally  the  training  would  permanently  change  people’s  behavior,  and  the  behaviors  we  trained 
would  persist  over  time.  Future  experiments  in  which  the  effects  of  IM  training  are  assessed  over 
time  would  address  this  issue. 

4.3.2  Evaluation  Materials 

A  sensitive  and  thorough  evaluation  of  a  training  program  is  needed  to  assess  the  degree  to 
which  it  is  effective,  and  also  to  pinpoint  weaknesses  in  the  program,  identify  aspects  that  could  be 
strengthened,  and  suggest  possible  additions.  The  quality  of  the  evaluation  depends  upon  the 
scenario  materials  that  are  used,  the  evaluation  instruments  that  are  developed,  and  on  the 
capabilities  of  the  simulator  that  is  used  to  generate  the  scenario  and  to  capture  subjects’  actions 
and  responses  during  the  evaluation.  Based  on  the  observations  and  analyses  made  during  this 
project,  we  have  identified  improvements  that  could  be  made  to  the  scenario  materials  and  the 
simulator,  as  well  as  possible  changes  to  the  evaluation  procedure. 

Enhancements  to  Scenario  Materials 

When  the  amount  of  time  available  for  training  is  limited,  it  is  often  challenging  to  fit  in  all 

the  components  of  the  training  and  training  evaluation.  One  of  the  factors  that  often  falls  short  is 

the  amount  of  time  available  to  become  acquainted  with  the  names  of  places,  roads,  and 

geographic  features  of  the  locale  in  which  the  scenario  is  set.  For  example,  many  participants  were 

not  familiar  with  the  geographic  area  for  the  scenario  used  in  this  project,  and  felt  they  needed 

more  preparation  time  to  become  familiar  with  the  locations  of  key  areas  of  interest.  A  suggestion 

made  by  one  of  the  participants  is  that  the  scenario  could  be  set  in  a  location  that  is  familiar  to 

most  Army  officers  -  for  example,  the  National  Training  Center  in  California.  Using  familiar 

terrain  would  reduce  the  amount  of  time  needed  to  become  familiar  with  the  map  features. 
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As  we  noted  in  Section  4.1,  several  of  the  IM  errors  described  in  Figure  4-1  could  not  be 
detected  with  the  existing  scenario.  For  future  evaluations,  specific  ‘information  gaps’  could  be 
deliberately  planted  in  the  scenario  to  assess  whether  the  training  helped  participants  to  recognize 
these  gaps.  The  scenario  could  be  designed  to  include  a  sequence  of  related  but  nonrelevant 
messages.  This  could  be  used  to  assess  whether  trained  participants  are  less  likely  to  pursue 
nonrelevant  information,  for  example  by  tracking  down  activities  occurring  at  locations  that  are 
not  relevant  to  their  mission.  A  scenario  could  also  be  constructed  to  emphasize  critical  time 
windows.  It  is  difficult  to  incorporate  a  test  for  every  possible  kind  of  error  in  a  single  scenario. 
When  time  permits,  an  evaluation  involving  several  different  scenarios  would  be  advantageous. 

Enhancements  to  the  Simulator 

We  identified  several  modifications  that  would  improve  the  flexibility  and  capability  of  the 
simulator.  These  suggestions  are  based  on  observations  made  by  the  experimenters  and  comments 
made  by  the  participants. 

•  Provide  the  capability  to  delete  messages  from  the  queue.  This  is  a  standard  email  program 
capability  that  was  not  implemented  in  the  scenario  generator  we  used.  Providing  it  would 
allow  for  a  clearer  differentiation  between  messages  subjects  decided  were  nonrelevant  and 
messages  they  didn’t  have  sufficient  time  to  open. 

•  Provide  the  capability  for  multiple  recipients  of  messages.  Unlike  typical  email  programs,  the 
scenario  generator  did  not  permit  users  to  have  multiple  recipient  of  a  message.  This  is  a 
realistic  requirement  that  could  be  added,  and  is  important  for  assessing  the  aspects  of  IM 
training  concerned  with  knowledge  of  the  organization. 

•  Provide  the  capability  to  file  messages  into  specific  folders.  This  is  a  capability  available  in 
most  email  programs,  and  one  that  several  participants  requested.  Adding  it  to  the  scenario 
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generator  would  allow  the  experimenters  to  obtain  data  on  how  information  managers  group 
messages  they  receive,  and  how  they  applied  knowledge  of  the  organization. 

•  Provide  an  easy  mechanism  to  send  messages  to  recipients  that  are  not  on  a  prespecified  list. 
At  present  messages  can  only  be  sent  to  nodes  on  a  predefined  list.  Adding  this  feature  would 
allow  for  better  assessment  of  aspects  of  training  concerned  with  information  transmission. 

Changes  to  the  Evaluation  Procedure 

A  number  of  participants  commented  that  the  scenario  procedure  was  rather  unrealistic. 
Although  subjects  could  send  messages  requesting  information  or  directing  actions  to  be 
performed,  they  did  not  receive  much  feedback  on  these  requests,  in  large  measure  because  of  the 
need  to  maintain  experimental  control  across  subjects.  The  officers  who  participated  in  the 
training  suggested  that  in  future  administrations  the  participants  should  have  a  more  active  role 
and  be  more  than  passive  information  managers.  It  would  also  be  more  realistic  and  more 
interesting  to  have  some  messages  delivered  via  different  media.  This  requires  a  scenario  that  is 
more  interactive,  involving  a  situation  with  multiple  participants.  To  maintain  experimental 
control,  it  may  be  possible  to  have  some  of  the  roles  played  by  confederates  rather  than  other 
participants. 

4.4  CONCLUSION 

The  goals  of  the  current  program  were  to  develop  a  theory-based  model  of  information 
management,  and  from  that  to  develop  a  training  program  that  can  be  used  to  train  effective 
information  management.  As  is  documented  in  this  report,  this  program  did  just  that;  a  training 
program  focused  on  the  specific  behaviors  “prepare,  filter,  scan,  read,  and  act”  did  help 
information  managers  deal  more  effectively  with  large  amounts  of  information.  In  the  preceding 
section  we  discussed  ways  in  which  we  can  improve  upon  our  training  paradigm  and  develop 
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it  even  further.  The  program  we  have  developed  fits  in  a  niche  that  has  been  overlooked  by 
researchers.  There  are  training  programs  intended  to  help  decision  makers  make  better  decisions 
(e.g.,  SHAPE;  Hartel  &  Hartel,  1997);  however,  none  of  these  other  training  programs  focus 
exclusively  on  the  process  necessary  to  manage  and  integrate  large  amounts  of  information  -  a 
process  Shattuck  and  his  colleagues  call  “cognitive  integration”  (Shattuck,  Graham,  Merlo,  and 
Hah,  2000)  -  and  the  specific  behaviors  needed  to  do  that.  Until  automatic  filtering  systems 
improve  dramatically  enough  to  make  the  need  for  human  filtering  obsolete,  a  training  program 
that  focuses  on  specific  behaviors  such  as  Prep,  Filter,  Scan,  Read,  and  Act  can  help  people 
manage  large  amounts  of  information  more  effectively.  Not  only  can  people  learn  to  better 
recognize  and  focus  on  important  information,  but  they  can  leam  to  change  behavior  in  such  a  way 
that  they  no  longer  contributes  to  other  people’s  overload. 
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Information  Management  in  Distributed  Organizations 
Introduction  to  the  Study 

Under  the  sponsorship  of  the  Army  Research  Institute,  ALPHATECH,  Inc.  and  APTIMA, 
Inc.  are  conducting  a  research  project  exploring  information  management  in  distributed 
organizations.  The  Systems  Engineering  Department  at  the  Military  Academy  is  supporting  us  in 
this  project  and  has  given  us  permission  to  send  you  this  packet  of  materials  and  ask  for  your 
participation. 

You  will  participate  in  this  study  in  three  class  periods  during  the  week  beginning 
November  1 .  At  the  beginning  of  the  first  class  period  on  Monday,  we  will  ask  you  to  sign  a 
Consent  Form  indicating  that  you  are  willing  to  participate  in  this  study  and  we  will  administer  a 
brief  quiz  to  determine  your  retention  of  the  materials  in  this  package  (explained  below).  After  this 
initial  paperwork,  the  first  two  class  periods  (Monday  and  Tuesday)  will  focus  on  training  to 
prepare  you  for  the  scenario  -  which  includes  training  on  the  simulator  that  you  will  use  for  the 
experiment. 

During  the  third  class  period  (Wednesday,  Thursday,  or  Friday)  you  will  participate  in  an 
experiment  designed  to  help  us  explore  information  management  in  distributed  organizations. 
During  this  experiment  you  will  play  the  role  of  a  Joint  Force  Commander  who  must  develop  an 
assessment  of  the  situation  based  on  a  written  situation  brief  and  a  series  of  written  or  verbal 
update  messages  received  through  e-mail.  During  the  experiment  you  will  be  asked  to  give  two 
situation  briefings  (SITREPs),  one  at  the  midpoint  of  the  experiment  and  one  at  the  end.  At  the 
midpoint  you  will  deliver  your  situation  briefing  orally,  and  at  the  end  you  will  deliver  both  a 
written  and  an  oral  briefing.  At  the  end  of  the  experiment  on  the  third  day,  we  will  give  you  a  post 
experiment  questionnaire  that  we  ask  you  to  complete  and  return  to  Mai.  Kewley  in  class  Tuesday, 
9  November. 

We  are  asking  you  to  spend  approximately  one  to  one  and  a  half  hours  reading  through  two 
booklets  that  are  included  in  this  package  before  you  come  to  the  class  on  Monday.  The  first  one 
describes  the  background  situation  for  the  scenario  that  will  be  used  in  the  experiment.  It  is  entitled 
‘The  Road  to  War”.  The  second  one,  titled  “Gaining  Organizational  Knowledge”,  describes  the 
military  organization  that  is  used  in  this  scenario.  To  prepare  for  the  first  class  period,  we  ask  that 
you  look  over  the  scenario  and  carefully  study  the  material  explaining  the  organizational  structure. 

It  is  important  that  you  understand  the  organization  in  which  you  will  operate.  Don’t  be 
intimidated  by  the  bulk  of  the  two  handouts.  They  contain  a  number  of  maps,  tables,  and  charts 
that  take  up  a  good  amount  of  space  on  the  page  without  adding  much  “read  time.”  Please  bring 
these  materials  with  you  to  both  training  sessions  and  the  experiment. 

The  data  that  we  are  gathering  will  be  used  for  research  purposes  only.  Your  participation 
in  this  study  will  help  us  determine  how  people  are  affected  by  high  levels  of  information,  and 
what  can  be  done  to  improve  performance  in  times  of  high  information  load.  Because  we  want  to 
obtain  unbiased  responses  from  each  participant  in  the  study,  we  ask  that  you  not  discuss  this 
study  with  other  cadets  until  everyone  has  turned  their  materials  in  to  Maj.  Kewley  on  Nov.  9. 

We  thank  you  in  advance  for  supporting  this  study. 
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COMBINED  TASK  FORCE:  SCENARIO  PREPARATION 

Rationale:  Anytime  you  become  involved  with  a  new  organization,  it  is  essential  that  you  take 
the  time  to  get  to  know  each  person.  Specifically,  you  need  to  know  who  the  people  in  the  new 
group  or  organization  are  and  how  they  are  “connected”  to  the  other  people  in  the  organizaiton. 
This  is  sage  advice  for  everyday  life,  as  well  as  for  your  participation  in  the  coming  study.  To 
prepare  for  this  mission  you  must  arrive  at  the  study  sight  with  a  thorough  understanding  of  the 
organization  being  used  in  the  study.  Following  is  an  organizational  chart  and  a  general 
description  of  the  primary  communications  nodes  (i.e.,  people)  involved  in  the  study.  Upon  your 
arrival  at  the  session  you  will  be  tested  to  ensure  you  have  sufficient  knowledge  of  the 
organization. 
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Organizational  Chart 


Normally  when  you  join  a  new  organization,  you  receive  a  chart  like  the  one  above.  For  the  upcoming 
study,  we  want  you  to  be  able  to  focus  on  the  study  instead  of  having  to  learn  about  the  people  and 
their  functions.  For  this  purpose  we  have  devised  this  booklet  with  explanations  of  the  people  in  the 
organization. 
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CINC,  MEDCOM:  Commands  MEDCOM  (which  includes  Algeria  &  Tunisia). 

CTFC:  Tasked  with  three  phased  operation  to  destroy  Algerian  forces  in  Tunisia 

JOC:  Operations  staff  for  CTFC.  Monitors  and  deconflicts  operations  carried  out  by  the 
component  commanders. 

JIC:  Intelligence  staff  for  CTFC.  Collect,  analyze,  and  disseminate  enemy  information. 

JFLC:  Clearing  house  for  components'  logistic  requests,  (e.g.,  food,  fuel,  ammo,  transport). 
CFACC:  Allocation  and  control  of  all  air  assets  (including  missiles)  assigned  to  CTF  Agile. 
CFACC  Ops  Ctr:  Provides  support  to  the  component  commander  that  JOC  provides  to  CTFC. 
CFACC  Intel  Ctr:  Provides  support  to  the  component  commander  that  JIC  provides  to  CTFC. 
JFSC:  Central  coordination  for  fire  support  requests. 

CFLCC:  Tasking,  control,  and  coordination  of  all  ground  force  units  (non-SOF)  ashore  during 
ground  operations. 

CFLCC  Ops  Ctr:  Provides  support  to  the  component  commander  that  JOC  provides  to  CTFC. 
CFLCC  Intel  Ctr:  Provides  support  to  the  component  commander  that  JIC  provides  to  CTFC. 
CFMCC:  Tasking,  control,  and  coordination  of  all  joint  naval  forces  assigned  to  CTF  Agile. 
CFMCC  Ops  Ctr:  Provides  support  to  the  component  commander  that  JOC  provides  to  CTFC. 
CFMCC  Intel  Ctr:  Provides  support  to  the  component  commander  that  JIC  provides  to  CTFC. 
Cmdr  FORCE  I:  Sever  enemy  logistic  support  lines. 

Cmdr  FORCE  II:  Operate  against  5th  Infantry  Division,  interdict  coastal  road. 

Cmdr  FORCE  III:  (Operations  Deception):  Conduct  deception  operations. 

Cmdr  FORCE  IV:  Seize  Sfax,  and  destroy  Algerian  forces  in  Tunisia. 

CFSOCC:  Tasking,  control,  and  coordination  of  all  joint  force  special  operations  units. 

CFSOCC  Ops  Ctr:  Provides  support  to  the  component  commander  that  JOC  provides  to 
CTFC. 

CFSOCC  Intel  Ctr:  Provides  support  to  the  component  commander  that  JIC  provides  to  CTFC. 
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TASK  LOAD  INDEX  (TLX)  WORKLOAD  QUESTIONNAIRE 


INSTRUCTIONS 

Workload  is  a  difficult  concept  to  define  precisely,  but  a  simple  one  to  understand 
generally.  The  factors  that  influence  your  experience  of  workload  may  come  from  the 
task  itself,  your  feelings  about  your  own  performance,  how  much  effort  you  put  in,  or  the 
stress  and  frustration  you  felt.  The  workload  contributed  by  different  task  elements  may 
change  as  you  become  more  familiar  with  a  task,  perform  easier  or  harder  versions  of  it, 
or  move  from  one  task  to  another.  Physical  components  of  workload  are  relatively  easy 
to  conceptualize  and  evaluate.  However,  the  mental  components  of  workload  are  more 
difficult  to  measure,  so  we  offer  the  following  discussion  to  help  you  with  the  problem. 

Workload  is  something  that  is  experienced  individually  by  each  person.  There  are 
no  effective  "rules"  that  can  be  used  to  estimate  the  workload  of  different  activities.  One 
way  to  find  out  about  workload  is  to  ask  people  to  describe  the  feelings  they  experienced. 
Because  workload  may  be  caused  by  many  different  factors,  we  will  ask  you  to  evaluate 
each  of  the  component  factors  individually  rather  than  lumping  them  into  a  single  global 
evaluation  of  overall  workload. 

The  six  rating  scales  used  in  this  workload  questionnaire  were  developed  for  you 
to  use  in  evaluating  your  experience  during  a  particular  segment  of  the  experiment. 

Please  read  the  descriptions  of  the  scales  below  carefully.  If  you  have  a  question  about 
any  of  the  scales,  you  can  ask  them  now  or  you  can  hold  your  questions  until  the  first 
workload  assessment  period  and  ask  them  at  that  time,  in  the  context  of  the  task  you  have 
just  performed.  It  is  important  that  the  scales  are  clear  to  you.  The  descriptions  of  the 
scales  will  always  be  available  to  you  when  you  do  your  workload  ratings.  Ask  if  you 
wish  to  review  them. 

You  will  be  asked  to  fill  out  a  workload  questionnaire  after  you  complete  each  set 
of  experiment  trials.  Please  respond  to  each  of  the  six  rating  scales  in  terms  of  your 
experiences  for  that  set  of  trials  by  putting  an  "X"  at  the  point  on  the  scale  that  matches 
your  experience.  Each  scale  has  two  endpoint  descriptors  that  describe  the  scale.  In  five 
of  the  six  scales  the  end  points  for  the  scale  are  “very  low”  (on  the  left)  and  “very  high” 
(on  the  right).  Note  that  "performance"  goes  from  "perfect"  on  the  left  to  "failure"  on  the 
right.  This  order  has  been  confusing  for  some  people. 


Please  consider  your  responses  carefully  in  distinguishing  among  the  different 
task  conditions.  Consider  each  scale  individually.  Your  accurate  ratings  will  play  an 
important  role  in  the  evaluation  being  conducted. 


DEFINITION  OF  THE  TLX  SCALES 


The  mental  demand  scale  asks  you  to  rate  how  much  mental  and 
perceptual  activity  was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.).  Was  the  task  or  situation  easy  or 
demanding,  simple  or  complex,  exacting  or  forgiving?  Make  this  rating 
on  a  scale  from  very  low  mental  demand  to  very  high  mental  demand. 

The  physical  demand  scale  asks  you  to  rate  how  much  physical  activity 
was  required  (pushing,  pulling,  turning,  controlling,  activating,  etc.). 

Was  the  task  or  situation  easy  or  demanding,  slow  or  brisk,  slack  or 
strenuous,  restful  or  laborious?  Make  this  rating  on  a  scale  from  very 
low  physical  demand  to  very  high  physical  demand. 

The  temporal  demand  scale  asks  you  to  rate  how  much  time  pressure 
you  felt  due  to  the  rate  or  pace  at  which  the  task  or  task  elements 
occurred.  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic?  Make 
this  rating  on  a  scale  from  very  low  temporal  demand  to  very  high 
temporal  demand. 

The  performance  scale  asks  how  successful  you  think  you  were  in 
accomplishing  the  goals  of  the  task  or  situation  set  by  the  mission  (or 
yourself).  How  satisfied  were  you  with  your  performance  in 
accomplishing  these  goals?  Make  this  rating  on  a  scale  from  perfect 
(successfully  accomplished  everything)  to  failure  (nothing  was 
successfully  accomplished). 

The  effort  scale  asks  you  to  rate  how  hard  you  had  to  work  (mentally 
and  physically)  to  accomplish  your  level  of  performance.  Make  this 
rating  on  a  scale  from  very  low  effort  (not  hard  at  all)  to  very  high  effort 
(very  hard). 

The  frustration  level  scale  asks  how  insecure,  discouraged,  irritated, 
stressed,  and  annoyed  versus  secure,  gratified,  content,  relaxed,  and 
complacent  you  felt  during  the  situation  that  you  just  experienced. 

Make  this  rating  on  a  scale  from  very  low  frustration  to  very  high 
frustration. 


TLX  WORKLOAD  QUESTIONNAIRE 

SUBJECT  ID: _ _  DATE: _ 


PLEASE  RATE  THE  WORKLOAD  YOU  EXPERIENCED  ON  THE  SIX  SCALES 
BELOW  BY  PUTTING  AN  "X"  ON  EACH  OF  THE  SCALES  AT  THE  POINT  THAT 
MATCHES  YOUR  EXPERIENCE 


Mental  Demand 


Very  Low 


Very  High 


Physical  Demand 


Very  Low 


Very  High 


Temporal  Demand  I  _ I  1  I  I  I  I  I  I  I 

(Time  Pressure)  Very  Low  Very  High 


Performance 


Perfect 


Failure 


Effort 


Very  Low 


Very  High 


Frustration 


Very  Low 


Very  High 
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BRIEFING  PREPARATION  FORM 

Briefly  answer  the  questions  below  in  writing.  Doing  this  will  help  you  develop 
and  focus  your  oral  SITREP. 

1 .  What  is  the  most  critical  uncertainty  right  now  and  how  are  you  trying  to  resolve  it? 


2.  What  information  would  you  like  to  have  right  now  to  get  a  better  assessment  of  the 
situation?  Where  (or  who)  will  you  try  and  get  it  from? 


3.  What  predictions  can  you  make  based  on  what  you  know  now  as  to  how  the  situation 
may  evolve? 


4.  What  is  your  hypothesis  at  this  time  to  explain  how  this  situation  occurred? 
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Rater’s  initials 


Subject  Number 


Situation  Assessment  Evaluation  Scale 


A:  Assessment  of  important  themes:  One  point  for  each  topic/theme  mentioned. 


Topic/Theme 

Check  (/)if 
mentioned 

Armored  column  vicinity  of  objective  A  stopped  to  wait  for  fuel  (or  fuel 
trucks) 

Algerian  armed  forces  experiencing  difficulty  moving  fuel  to  its  units  - 
fuel  trucks  broken  down  or  destroyed;  or  resorting  to  using  unmarked 
or  “relief  agency”  marked  trucks  to  move  fuel. 

Armored  column  personnel  (in  vicinity  of  Objective  A)  do  not  appear  to 
be  digging  in  or  taking  up  defensive  position;  or,  personnel  appear  to 
be  milling  around. 

Enemy  gunships  (MI  -  8  Helos)  in  vicinity  of  Objective  1  are  arming  and 
fueling  -  area  may  be  “hot” 

Some  SCUD  and  SAM  launchers  still  operational ;  or,  enemy  using 
decoys  to  simulate  missile  sites 

Algerian  forces  no  longer  moving  into  Tunesia  -  Algerian  forces  appear 
to  be  massing  in  preparation  to  repel  counter  invasion. 

Total  (add  check  marks) 

B:  Overall  accuracy  of  Situation  Assessment:  Using  the  attached  sheet  showing  the 
information  and  instructions  given  to  the  subjects,  and  the  ‘ground  truth’  provided  by  our 
subject  matter  expert  who  developed  this  scenario,  circle  the  number  that  best  corresponds  with 
the  cadet’s  overall  accuracy  of  the  situation  assessment;  use  the  provided  anchors  to  help  choose 
the  appropriate  score. 


1  2  3  4  5  6  7 


0)  _ 

Has  little  or  no  understanding 
of  the  situation,  hypotheses 
held  about  the  situation  have 
no  relationship  to  ground  truth 
(e.g.,  fails  to  mention  anything 
pertaining  to  Objective  1  or 
Objective  A.  Focuses  on 
unrelated  occurrences  or 
simply  says  “I  don’t  know 
what’s  going  on”) _ 


_ (4} _ 

Has  limited  understanding  of 
the  situation,  hypotheses  held 
have  some  similarities  with, 
and  some  differences  from, 
ground  truth  (e.g.,  believes 
that  the  armored  column  has 
run  out  of  gas,  and  that  it  also 
poses  a  threat  to  Objective  1) 


_ (21 

Has  very  accurate 

understanding  of  the  situation, 
hypotheses  held  are  very  close 
to  or  exactly  ground  truth 
(e.g.,  Seems  to  know  that  the 
armored  column  poses  no 
threat  because  it  is  simply 
waiting  for  fuel) 


Comments/Notes  on  your  overall  rating: 
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POST-STUDY  QUESTIONNAIRE 

NAME: _  DATE: _  CLASS  TIME: _ 

Please  answer  each  question  below.  Briefly  respond  to  the  open-end  questions.  Please  answer  as 
honestly  as  you  can;  by  doing  so,  you  will  help  us  refine  the  training  to  make  it  more  effective.  Turn  the 
completed  questionnaire  in  to  Maj.  Kewley  in  class  9  November  1999.  THANK  YOU! 


1.  After  reading  the  “Road  to  War  and  Situation  Brief,”  how  clear  was  the  current  situation? 

1  1  I  2  I  3  I  4  I  5  I  6  I  7 

not  very  clear  midpoint  very  clear 

2.  After  you  read  the  OP  order  prior  to  the  beginning  of  the  trial,  how  clear  was  what  you  had  to  do? 

i  1 1  2 1  3 1  4 1  5 1 6  i 7  i 


not  very  clear  midpoint  very  clear 

3.  When  you  began  the  trial,  how  clear  were  you  about  how  to  use  the  e-mail  message  software? 

7  I 


1  i  2  i  3  i  4 


5  j  6 


not  very  clear 


midpoint 


very  clear 


4a.  Prior  to  the  start  of  the  trial,  did  you  have  sufficient  time  to  consider  the  information  and  formulate  a 
assessment  of  the  situation? 


1  1 

i  2 1 

3  ! 

1  4  |  5  |  6 

i . 7 

not  enough  time 

midpoint 

plenty  of  time 

4b.  During  the  trial,  did  you  have  sufficient  time  to  consider  the  information  and  formulate  a  hypothesis 

to  what  was  occurring  in  the  situation? 

i  1  j 

2 1 

3  1 

. 4  i  5  i  6 

I  7  1 

not  enough  time 

midpoint 

plenty  of  time 

5a.  How  adequate  was  the  time  allotted  to  prepare  your  oral  briefings? 

1 1 

2 1 

3  1 

4  i  5  i  6 

i  7  1 

very  inadequate 

midpoint 

very  adequate 

5b.  How  adequate  was  the  time  allotted  to  prepare  the  final  written  brief? 

i  1  i 

2 1 

3  1 

4  i  5  i  6 

I  7  1 

very  inadequate 

midpoint 

very  adequate 

6a.  What  percentage  of  the  messages  did  you  leave  unopened  by  choice? 


0%  25%  50%  75%  100% 

6b.  What  criteria  did  you  use  to  recognize  messages  to  be  left  unopened  by  choice? 
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7.  What  percentage  of  the  messages  did  you  leave  unopened  because  you  did  not  have  time  to  open  them 

I _ I _ I _ I _ I _ I _ I _ | _ | 

0%  25%  50%  75%  100% 

8a.  What  percentage  of  the  written  messages  did  you  only  skim  and  not  read  carefully? 

L _ _ L . 1 _ I _ I _ I _ I _ I _ I 

0%  25%  50%  75%  100% 

8b.  What  criteria  did  you  use  to  know  that  skimming  would  be  satisfactory? 


9a.  If  you  heard  auditory  messages,  what  percentage  of  the  auditory  messages  did  you  leave  unopened  by 
choice? 

I _ 1 .  I _ I _ L_ _ I _ I _ I _ I 

0%  25%  50%  75%  100% 

9b.  If  you  heard  auditory  messages,  what  percentage  did  you  play  more  than  once? 

1  1 _ I _ | _ | _ | _ | _ | _ | 

0%  25%  50%  75%  100% 

9c.  How  did  you  know  (determine)  which  one(s)  to  replay? 


9d.  What  was  the  principal  reason  you  replayed  a  message? 

10a.  If  you  heard  auditory  messages,  to  what  extent  would  you  prefer  them  over  written  messages? 

I _ I _ I _ I _ I _ I 

very  little  midpoint  a  lot 

10b.  If  you  did  not  hear  auditory  messages,  to  what  extent  do  you  think  you  would  prefer  verbal  messages 
over  written  messages? 

I _ I _ I _ I _ l__ _ I 

very  little  midpoint  a  lot 

11.  What  criteria  did  you  use  to  prioritize  messages  as  to  their  criticality? 


12a.  How  helpful  was  the  information  about  organizational  structure  to  you? 


not  very  helpful  midpoint  very  helpful 

12b.  In  what  way  was  the  information  about  organizational  structure  helpful  to  you? 


13a.  How  much  information  did  you  forward/send  to  other  nodes? 


very  little 


midpoint 


a  great  deal 
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13b.  How  difficult  was  it  to  determine  to  what  node  to  send  information? 


1 

i  2 

i  3  i  4  i  5  i 

6  i 

7 

very  difficult 

midpoint 

not  very  difficult 

13c.  What  criteria  did  you  use  to  decide  what  information  to  send? 


13d.  If  you  did  not  send  information  to  other  nodes,  why  not? 


14a.  How  much  information  did  you  request  from  other  nodes? 

1  1  i  2  i  3  i  4  i  5  i  6  i  7  1 

very  little  midpoint  a  great  deal 

14b.  How  difficult  was  it  to  determine  from  what  node  to  request  information? 


1 

i  2 

i  3  i  4  i  5  i 

6  I 

7 

very  difficult 

midpoint 

not  very  difficult 

14c.  What  criteria  did  you  use  to  decide  what  information  to  request  from  a  node? 


14d.  If  you  did  not  request  information  from  other  nodes,  why  not? 


15.  In  general,  what  were  the  most  important  factors  you  considered  when  formulating  your  briefing? 


16a.  What  is  your  overall  rating  of  the  written  training  materials? 


1  |  2  |  3 

i  4  i  5  i 

6  I 

7 

very  poor 

midpoint 

very  good 

16b.  What  is  your  overall  rating  of  the  “in-class” 

training  activities? 

i 1  i 2  i 3 

i  4  i . 5  i 

6  1 

7  1 

very  poor 

midpoint 

very  good 

17a.  To  what  extent  did  the  skills  you  learned  in 

training  helped  you  remain  focused  on  the  mission? 

i  ■  i 2  i 3 

i  4  i  5  i 

6  1 

7 

very  little 

midpoint 

a  great  deal 

17b.  Please  explain  give  an  example. 


18.  Any  general  comments  about  the  study? 
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SAMPLE  MESSAGES 


Critical  Messages 


From  :CTF_AGILE_INTELLIGENCE 
Subject:  INTEL  UPDATE 
DTG:  092330L  SEP  08 

Message  Text: 

SIGINT  INTERCEPT  INDICATES  UNITS  ASSOCIATED  WITH  THE  3RD  CAA  3ro 
MECHANIZED  INFANTRY  DIVISION  TRYING  TO  CONTACT  REFUELING  TRUCKS.  NO 
RESPONSE. 

From  :CTF_AG  ILEJNTELLIGENCE 
Subject:  INTEL  UPDATE 
DTG:  092200L  SEP  08 

Message  Text: 

FUSED  SOURCES  INDICATE  ALGERIANS  ATTEMPTING  TO  REINFORCE  TABARKA. 
INDICATE  CRITICAL  LOGISTICS  NODE  STILL  EXISTS  AT  TABARKA  DESPITE 
REPEATED  AIRSTRIKES.  RAILS  SEVERED  IN  STRIKES  ON  2  SEP  APPEAR  TO  BE 
REPAIRED. 


Nonrelevant  Messages 


From:  MEDIA_SOURCES 
Subject:  MEDIA  REPORTS 
DTG:  092240 L  SEP  08 

Message  Text: 

MEDIA  REPORTS  INDICATE  THAT  US  AIR  STRIKES  NEAR  KELAA-ES-SENAM  HAVE 
RESULTED  IN  THE  DESTRUCTION  OF  A  RELIEF  COLUMN  MOVING  FOOD,  MEDICAL 
SUPPLIES,  AND  DIESEL  FUEL  TO  A  REFUGEE  CAMP  IN  REBIB.  12  PEOPLE  WERE 
REPORTED  KILLED  AND  6  INJURED.  DETAILS  ARE  SKETCHY. 

From:  CTF_AGILEJNTELLIGENCE 
Subject:  INTELL  UPDATE 
DTG:  092330L  SEP  08 

Message  Text: 

HUMINT  REPORTS  INDICATED  GROUP  OF  FUEL  AND  COMMERCIAL  TRUCKS  WITH 
RELIEF  AGENCY  MARKINGS  TURNED  OFF  ROAD  49  AND  ONTO  ROAD  53  AT 


APPROXIMATELY  2100L  NEAR  OUN  TEBOUL,  ALGERIA.  TRUCKS  WERE  MOVING 
AT  HIGH  SPEED. 
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APPENDIX  B 


Background  Questionnaire 


Name _  Date _ Time 


The  purpose  of  this  questionnaire  is  to  obtain  some  information  about  your  military 
background  and  experiences.  This  information  will  be  used  to  better  understand  your 
responses.  All  information  collected  will  remain  confidential  and  will  not  be  released  to 
third  parties.  We  appreciate  your  cooperation  in  completing  this  form. 


Current  Status:  _ Active  Duty 

_ Retired;  Date  of  separation 


Rank 

Branch 

Specialty 

Time  in  grade 

Last  service  school  attended 

Time  in  Service  _ 

Year 

_ years 

How  many  exercises  have  you  participated  in  as  a  G-3  or  S-3 
What  was  the  last  one? 


Name: _  Date/Time: _ 

Decision  Making  in  Distributed  Organizations 

Post-training  questionnaire 

As  quickly  as  possible,  please  answer  the  following  questions.  Please  make  your  answers 
brief  and  concise. 

1)  What  is  one  way  the  Army  of  the  future  will  be  different  from  that  of  today? 


2)  You  have  just  been  charged  with  monitoring  message  flow  to  help  handle  a  very 
complicated  situation,  there  are  56  messages  in  the  inbox;  briefly  describe  how  you 
should  proceed. 


3)  Based  on  your  knowledge  of  “typical”  information  exchanges  during  wartime: 

a)  what  percentage  of  those  56  messages  in  Item  #  2’s  inbox  do  you  think  would  be 
critical  for  your  task? 

b)  What  percentage  would  be  irrelevant? 

4)  What  is  a  technique  you  can  use  to  decide  if  a  message  is  relevant  for  your  task? 


5)  How  is  the  structure  of  the  organization  used  in  this  research  different  from  the 
organization  of  today? 
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BRIEFING  PREPARATION  FORM 

Briefly  answer  the  questions  below  in  writing.  Doing  this  will  help  you  develop 
and  focus  your  oral  SITREP. 

1 .  What  is  the  most  critical  uncertainty  right  now  and  how  are  you  trying  to  resolve  it? 


2.  What  information  would  you  like  to  have  right  now  to  get  a  better  assessment  of  the 
situation?  Where  (or  who)  will  you  try  and  get  it  from? 


3.  What  predictions  can  you  make  based  on  what  you  know  now  as  to  how  the  situation 
may  evolve? 


4.  What  is  your  hypothesis  at  this  time  to  explain  how  this  situation  occurred? 
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Rater’s  initials 


Subject  Number 


Situation  Assessment  Evaluation  Scale 

INTERIM  BRIEFING 


^^Assessmen^nm^ortan^hemes^Onepointfo^acl^opicAhemementioned. 


Topic/Theme 

Check  (/ )  if 
mentioned 

9th  Algerian  Mech  Division  increased  activity  (e.g.,  increased  alert  status,  conducting 
preparations  for  combat,  logistics  stockpiling,  equipment  maintenance,  rehearsal 
activity)  indicates  they  are  going  to  relocate  or  conduct  combat  operations 

Algerian  refueling  activities  implies  major  relocation,  not  just  minor  shifts 

Increased  security  at  the  command  post  suggests  major  planning  efforts  ongoing 

Movement  of  goods  West  to  East  across  Tunisia  suggests  movement  to  reinforce  forces 
vicinity  Tunis.  (Don’t  necessarily  have  to  believe  they  are  going  to  reinforce  9th  Div  in 
order  to  make  salient  point) 

Influx  of  barrier  materials  and  mines  to  the  9th  Algerian  Mech  Division  suggests  they 
will  conduct  defensive  operation 

Air  screen  to  the  south  of  NAIs  34  and  35  suggests  enemy  moving  south:  could  be  to 

Sousa,  or  perhaps  to  Sfax 

Enemy  movement  (e.g.,  APCs,  armor,  artillery)  around  El  Djam  and  south  of  NAIs  34 
and  35  toward  Sfax  suggesting  preparations  for  a  defense  against  forces  attacking  from 
the  south 

Algerian  order  to  conduct  defense  with  phased  movement  from  the  current  location  to 
the  defensive  sector  also  suggests  enemy  may  be  moving  south. 

Possible  compromise  of  CTF  intentions  through  Libyan  observations 

9th  Div  represents  a  threat  to  Force  IV  security  and  mission  accomplishment 

Total  (add  check  marks) 

B:  Overall  quality  of  Situation  Assessment:  Circle  the  number  that  best  corresponds  with  the 
subject’s  overall  accuracy  of  the  situation  assessment;  use  the  provided  anchors  to  help  choose  the 
appropriate  score. 


12  3  4  5  6  7 


m _ 

Has  little  or  no  understanding  of 
the  situation,  hypotheses  held 
about  the  situation  have  no 
relationship  to  ground  truth  . 
Briefing  only  reiterates 
information  given  in  the  Vignette 
(e.g.,  states  there  is  increased 
activity  in  9th  Algerian  Mech 
Div.,  says  deception  plan 
working  but  gives  no  evidence) 


_ (D _ 

Has  limited  understanding  of  the 
situation,  hypotheses  held  have 
some  similarities  with,  and  some 
differences  from,  ground  truth  . 
Goes  somewhat  beyond  stated 
information  in  integrating 
separate  messages,  suggesting 
implications,  etc.  Gives  some 
backup  for  conclusions,  (e.g., 
mentions  one  factor  that  indicates 
9th  Mech  may  be  moving  south.) 


_ (21 

Has  very  accurate  understanding 
of  the  situation,  hypotheses  held 
are  very  close  to  or  exactly 
ground  truth .  Understands 
implications  of  information  in 
messages.  Backs  up  all 
conclusions.  Suggests  next 
planning  steps  needed 
(e.g.,  draws  appropriate 
inferences  and  implications  from 
activities  south  of  NAI  34/35, 
describes  info/allocations  needed 
_ from  CTF  Agile). 


vl.2 


Rater’s  initials 


Subject  Number 


C:  How  well  did  subject  integrate  information  contained  in  messages:  Circle  the  number  that 
best  corresponds  with  the  subject’s  overall  accuracy  of  the  situation  assessment;  use  the  provided  anchors 
to  help  choose  the  appropriate  score. 


1  2 

3  4  5 

6  7 

All _ 

_ (4) _ 

(7) 

May  provide  individual  pieces  of 
information,  but  gives  no 
evidence  he/she  is  putting 
separate  pieces  of  information 
together 

Puts  together  information  from 
some  messages  and  projects 
implications  to  limited  extent 
(e.g.  sees  relationship  between 
logistics  movement  and  troop 
activities) 

Pulls  together  all  information 
provided  into  coherent  picture. 
States  critical  information  and 
appropriate  relationships  among 
different  pieces,  (e.g.  see 
relationship  between  activities  in 
Tunis,  9lh  Div  and  western 
LOCs). 

D:  To  what  extent  and  how  well  (appropriately)  did  subject  provide  the  means  to  improve 
SA  and  reduce  uncertainty:  Circle  the  number  that  best  corresponds  with  the  subject’s  overall 
accuracy  of  the  situation  assessment;  use  the  provided  anchors  to  help  choose  the  appropriate  score. 

1  2 

3  4  5 

6  7 

(1) 

(4) 

(7) 

Offered  no  suggestions,  or  only 
vague  notions,  for  where 
additional  or  supporting 
information  could  be  obtained. 

Makes  some  suggestions  for 
what  kinds  of  information  should 
be  sought  and  where  to  look  for 
it.  (e.g.,  requests  coverage  of  9th 
Mech  Div  or  movement  of 
forces;  suggests  one  possible 
alternative  for  obtaining  info) 

Makes  several  suggestions  for 
information  needed  and  where  it 
should  be  obtained  (e.g., 
establish  new  NAIs,  tasking 
JSTARS,  coordinate  w/  Force  IV 
G2  for  coverage  of  movement  of 
forces  from  9th  Mech  Div  AO 
north  and  south,  request  recon 
assets  from  higher).. 

Comments/Notes: 


vl.2 


Rater’s  initials 


Subject  Number 


Situation  Assessment  Evaluation  Scale 
Final  Briefing 


A^_Assessment_onmportant  themes:  One  point  for  each  topic/theme  mentioned. 


Topic/Theme 

Check  (/)  if 
mentioned 

Enemy  movement  (e.g.,  APCs,  armor,  artillery)  around  El  Djam  and  south  of  NAIs  34 
and  35  toward  Sfax  suggesting  preparations  for  a  defense  against  forces  attacking  from 
the  south 

Risk  to  Force  IV  operations  vicinity  Sfax  -  lack  of  supporting  ODA  team,  other 

Algerian  activities  vicinity  Sfax. 

Required  information  or  planning  steps  necessary  to  change  mission  to  new  objectives 
(identify  necessity  to  begin  planning  for  new  branch  of  Operation  AGILE,  identification 
of  key  factors  affecting  change  in  mission,  e.g.  ability  of  9th  affect  operations  to  north 
and  south) 

Movement  of  goods  West  to  East  across  Tunisia  suggests  movement  to  reinforce  forces 
vicinity  Tunis.  (Don’t  necessarily  have  to  believe  they  are  going  to  reinforce  9th  Div  in 
order  to  make  salient  point) 

Operations  security  (e.g.  steps  needed  to  avoid  allowing  Algerian  forces  to  detect 
intentions  of  CTF  or  Force  IV,  possible  compromise  pf  CTF  intentions  through  Libyan 
observations) 

Total  (add  check  marks) 

B:  Overall  quality  of  Situation  Assessment:  Circle  the  number  that  best  corresponds  with  the 
subject’s  overall  accuracy  of  the  situation  assessment;  use  the  provided  anchors  to  help  choose  the 
appropriate  score. 


1 _ 2 _  3  4  5  6  7 


iil_ _ 

Has  little  or  no  understanding  of 
the  situation,  hypotheses  held 
about  the  situation  have  no 
relationship  to  ground  truth  . 
Briefing  only  reiterates 
information  given  in  the  Vignette 
(e.g.,  states  there  is  increased 
activity  in  9th  Algerian  Mech 
Div.,  says  deception  plan 
working  but  gives  no  evidence) 


_ (A 1 _ 

Has  limited  understanding  of  the 
situation,  hypotheses  held  have 
some  similarities  with,  and  some 
differences  from,  ground  truth  . 
Goes  somewhat  beyond  stated 
information  in  integrating 
separate  messages,  suggesting 
implications,  etc.  Gives  some 
backup  for  conclusions,  (e.g., 
mentions  one  factor  that  indicates 
9th  Mech  may  be  moving  south.) 


_ 01 

Has  very  accurate  understanding 

of  the  situation,  hypotheses  held 
are  very  close  to  or  exactly 
ground  truth .  Understands 
implications  of  information  in 
messages.  Backs  up  all 
conclusions.  Suggests  next 
planning  steps  needed 
(e.g.,  reference  to  new  planning 
required,  describes 
info/allocations  needed  from 
_ CTF  Agile). 


VI  final  brief 


Rater’s  initials 


Subject  Number 


C:  How  well  did  subject  integrate  information  contained  in  messages:  Circle  the  number  that 
best  corresponds  with  the  subject’s  overall  accuracy  of  the  situation  assessment;  use  the  provided  anchors 
to  help  choose  the  appropriate  score. 


2  3  4  5  6  7 


(1) 

(4) 

(7) 

May  provide  individual  pieces  of 
information,  but  gives  no 
evidence  he/she  is  putting 
separate  pieces  of  information 
together 

Puts  together  information  from 
some  messages  and  projects 
implications  to  limited  extent 
(e.g.  sees  relationship  between 
logistics  movement  and  troop 
activities) 

Pulls  together  all  information 
provided  into  coherent  picture. 
States  critical  information  and 
appropriate  relationships  among 
different  pieces,  (e.g.  see 
relationship  between  activities  in 
Tunis,  9th  Div  and  western 
LOCs). 

D:  To  what  extent  and  how  well  (appropriately)  did  subject  provide  the  means  to  improve 
SA  and  reduce  uncertainty:  Circle  the  number  that  best  corresponds  with  the  subject’s  overall 
accuracy  of  the  situation  assessment;  use  the  provided  anchors  to  help  choose  the  appropriate  score. 

1  2 

3  4  5 

6  7 

(1) 

(4) 

(7) 

Offered  no  suggestions,  or  only 
vague  notions,  for  where 
additional  or  supporting 
information  could  be  obtained. 

Makes  some  suggestions  for 
what  kinds  of  information  should 
be  sought  and  where  to  look  for 
it.  (e.g.,  requests  coverage  of  9th 
Mech  Div  or  lack  of  coverage 
where  ODA  Tm  now  missing; 
suggests  one  possible  alternative 
for  obtaining  info) 

Makes  several  suggestions  for 
information  needed  and  where  it 
should  be  obtained  (e.g., 
establish  new  NAIs,  tasking 
JSTARS,  coordinate  w/  Force  IV 
G2  for  coverage  of  9th  Mech  and 
other  forces  affecting  new 
objectives,  request  recon  assets 
from  higher,  collection 
capabilities  vicinity  of  new 
objectives).. 

Comments/Notes: 


VI  final  brief 


ALPHATECH,  INC. 


APTIMA,  INC. 


NAME: 


POST-STUDY  QUESTIONNAIRE 
_  DATE:  TIME: 


Please  answer  the  questions  below  as  honestly  as  you  can;  by  doing  so,  you  will  help  us  refine  the  training 
to  make  it  more  effective.  THANK  YOU! 


Part  I.  The  scenario  and  your  task 

1.  After  receiving  the  Road  to  War  and  Situation  Briefing,  how  clear  was  the  current  situation? 

i  1 1  2 1  3 1  4 1  5 1 6  i 7  i 


not  very  clear  midpoint  very  clear 

2.  When  you  started  the  data  collection  trial,  how  well  did  you  understand  the  mission? 

7 


1 1  2 1  3 1  4 


L_i_L 


riot  well  at  all 


midpoint 


very  well 


4.  When  you  began  the  trial,  how  clear  were  you  about  your  task? 

1  1  i  2  i  3  i  4  i  5  i  6  i  7 

not  very  clear  midpoint  very  clear 

4.  When  you  began  the  trial,  how  clear  were  you  about  your  information  needs? 

1 1  2 1  3 1  4 1  5 1 6  i 7  i 


not  very  clear 


midpoint 


very  clear 


5.  During  the  trial,  did  you  have  sufficient  time  to  consider  the  information  and  formulate  a  hypothesis  as  to 
what  was  occurring  in  the  situation? 


1  1 

2 1 

3  i  4  i  5 

i  6 

i  7 

not  enough  time 

midpoint 

plenty  of  time 

6.  During  the  trial,  to  what  extent  were  you  distracted  by  information  you  did  not  need? 

1.2.  3.4.5.  6.7' 


3  i  4  i  5 


very  distracted  midpoint  not  distracted  at  all 

7.  In  general,  what  were  the  most  important  factors  you  considered  when  formulating  your  briefing? 
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Part  2:  Information  Processing 

8.  How  important  were  each  of  the  following  aspects  of  the  header  to  you  in  deciding  whether  to  open  a 
message? 

8a.  Source  (from): 


not  important  at  all  midpoint  extremely  important 

8b.  Time: 


not  important  at  all 


midpoint 


extremely  important 


8c.  Subject: 


1 


1 


not  important  at  all 


midpoint 


U 

extremely  important 


9.  What  percentage  of  the  messages  did  you  leave  unopened  by  choice? 

I _ I _ I _ I _ I _ I _ I _ I _ I 

0%  25%  50%  75%  100% 


9a.  What  criteria  did  you  use  to  recognize  messages  to  be  left  unopened  by  choice? 


10.  What  percentage  of  the  messages  did  you  leave  unopened  because  you  did  not  have  time  to  open 
them 


0%  25%  50%  75%  100% 

1 1 .  Of  the  messages  you  opened,  what  percentage  of  the  messages  did  you  only  scan  and  not  read 
carefully? 

I _ | _ j _ I _ 1  I _ | _ | _ | 

0%  25%  50%  75%  100% 

12.  What  criteria  did  you  use  to  know  that  scanning  was  satisfactory? 


13.  When  you  read  a  message,  how  difficult  was  it  to  integrate  the  information  with  other  messages  in 
order  to  develop  a  hypothesis  about  what  was  happening  in  the  situation? 


not  difficult  at  all  midpoint  extremely  difficult 

14.  What  criteria  did  you  use  to  evaluate  the  importance  of  messages? 


15.  How  helpful  was  the  information  about  organizational  structure  to  you  in  processing  the  messages? 

1  i  2  i  3  i  4  i  5  i  6  i  7 


not  very  helpful 


midpoint 


very  helpful 
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16.  How  did  you  use  the  information  about  the  organizational  structure  in  processing  the  messages? 


17.  How  clear  were  you  about  the  information  needs  of  other  nodes  in  the  organization? 


not  very  clear  midpoint  very  clear 

18.0f  the  messages  you  opened,  what  percentage  did  you  forward  without  any  changes  to  another  node? 


0%  25%  50%  75%  100% 

1 8a.  What  percentage  of  the  messages  you  opened  did  you  forward  with  changes  to  another  node 


0%  25%  50%  75%  100% 


19.  How  difficult  was  it  to  determine  to  what  node  to  send  information? 


1 

1  2 

i  3  i  4  i  5  i 

6  I 

7 

very  difficult 

midpoint 

not  very  difficult 

20.  What  criteria  did  you  use  to  decide  what  information  to  send? 


21 .  If  you  did  not  send  information  to  other  nodes,  why  not 

_ It  wasn’t  necessary  to  send  any  information  to  other  nodes 

_ Didn’t  have  enough  time 

_ Didn’t  know  where  to  send  the  information 

_ Didn’t  know  what  information  to  send 

_ Other:  Please  specify  )  _ _ 

22.  How  many  times  did  you  request  information  from  other  nodes? 

1  1  i  2  i  3  i  4  i  5  i  6  i  7  1 

never  midpoint  many  times 

23.  What  criteria  did  you  use  to  decide  what  information  to  request  from  a  node? 

24.  If  you  did  not  request  information  from  other  nodes,  why  not? 

_ Didn’t  have  enough  time 

_ Didn’t  need  any  information 

_ Didn’t  know  where  to  get  the  information 

_ Didn’t  know  what  to  ask 

_ Other:  Please  specify  ) _ 
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Part  3:  Training  Evaluation 

25.  How  relevant  was  the  information  in  the  training  program  for  you? 

i 1  i 2  i 3  i  *  i  5 1  ^  i  7 1 

not  relevant  at  all  midpoint  extremely  relevant 

26.  Did  you  learn  any  strategies  in  the  training  that  helped  you  cope  with  the  amount  of  information  in  the 
data  collection  scenario?  I  learned: 

_ None  _ 1  or  2  _ several. 

27.  Please  give  an  example  of  a  strategy  you  used.  _ __ 


28.  To  what  extent  do  you  think  this  training  is  applicable  to  dealing  with  large  volumes  of  information  in 
your  real  work? 


1 


not  at  all  midpoint 

29.  What  is  your  overall  rating  of  the  training  program? 

1  i  2  i  3  i  4  i 


i  4  i  5  i  6  i 


a  great  extent 


very  poor  midpoint 

30.  What  suggestions  do  you  have  for  improving  the  training? 


very  good 


3 1 .  Any  general  comments  about  the  study? 
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Sample  Messages 
Evaluation  Vignette 


Critical 


From:  G2,  CTF  AGILE 
To:  CDR,  FORCE  IV 
Subject:  INTEL  UPDATE 
DTG:09  1720L  SEP  08 
Message  Text: 

Current  estimate  of  9th  Algerian  Mech  Division  activities:  9th  Algerian 
Mech  Division  is  currently  at  an  increased  alert  status  and  conducting 
preparations  for  combat,  including  logistics  stockpiling,  equipment 
maintenance,  and  rehearsal  activity.  Activity  indicates  probable  change 
of  mission  and  likely  execution  within  12  to  48  hours. 

From:  JSTARS 
To:  CDR,  FORCE  IV 

Subject:  SPOT  REPORT:  APCs  and  tanks  moving  toward  El  Djam 
DTG:09  2100L  SEP  08 
Message  Text: 

Approximately  three  battalions'  worth  of  Enemy  APCs  and  tanks  are 
moving  southeast  toward  El  Djam  (35-1 8N  010-44E).  A  ground 
reconnaissance  squadron  has  established  security  and  traffic  control 
forward  of  the  movement.  There  is  also  an  increase  of  air  activity  in  an 
apparent  guard  mission  in  support  of  this  ground  movement. 

Relevant 


From:  FORCE  IV  Ml  (IEW  Unit) 

To:  CDR,  FORCE  IV 

Subject:  SIGINT  REPORT:  Increased  traffic  from  vicinity  of  9th  Algerian 

Mech  Division 

DTG:  09  1 700L  SEP  08 

Message  Text: 

SIGINT  has  detected  radio  increased  radio  communications  from  assumed 
command  and  logistics  nodes  in  the  vicinity  of  the  9th  Algerian  Mech 
Division.  There  has  been  a  corresponding  slight  increase  in  radio 


transmissions  from  nodes  in  Tunis.  Transmissions  are  coded  and  not  yet 
intelligible. 

From:  ODA  Team  1 
To:  CDR,  CTF  AGILE;  CDR,  FORCE  IV 
Subject:  SPOT  REPORT:  NAI  35 
DTG:  09  1 81 OL  SEP  08 

Message  Text: 

12  refuel  tankers  arrived  in  the  9th  Algerian  Mech  Division  support  area 
(NAI  35)  at  091730L  SEP  08.  Unit  tanks  and  mechanized  infantry 
vehicles  have  begun  refuel  operations. 

Nonrelevant 


From:  G3,  CTF  AGILE 
To:  CTF  AGILE 

Subject:  SITREP:  Additional  strikes  ordered  against  rail  LOCs 
DTG:  09  1820L  SEP  08 
Message  Text: 

Rail  LOCs  in  the  vicinity  of  Tabarka  have  been  redesignated  as  air 
strike  targets  for  the  next  12  hours. 

From:  G2,  CTF  AGILE 
To:  CTF  AGILE 

Subject:  INTREP:  Ship  Attack 
DTG:  09  2150L  SEP  08 

Message  Text: 

An  Algerian  Osaii  fired  two  SS-2  missiles  at  the  CVBG  operating  north 
of  Cap  Rosa.  Both  missiles  were  destroyed  in  the  air  by  the  USS  George 
Appleton,  an  Aegis  destroyer.  Both  the  Osaii  firing  the  missiles  and  the 
Osaii  in  company  were  reported  sunk  in  Tunisian  waters. 


